Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the application filed 11 August 2026. Claims 8 and 18 are cancelled. Claims 1, 5-7, 9-16, 19-23, and 25 are amended. Claims 1-7, 9-17, and 19-25 are pending and have been examined.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11 August 2026 has been entered.
Response to Arguments
Applicant's arguments, see pages 10-12, filed 13 July 2026, with respect to the rejections of Claims 1-7, 9-17, and 18-25 under 35 U.S.C. 101 have been fully considered are persuasive. The rejections of Claims 1-7, 9-17, and 18-25 under 35 U.S.C. 101 has been withdrawn.
APPLICANT'S ARGUMENT: Applicant argues (page 10, paragraph 6) that "Claim 1, for example, sets forth limitations whose net effect is a more efficient neural network. By adding and removing channels in a certain way (e.g., a specific number of post-addition channels) and by clipping among the pre- and post-addition channels based on importances of branches of a network block, the net effect is an improved neural network. Applicant's previous remarks have shown how this improvement occurs and how steps involved in the improvement are recited in the claims and reflected in the Specification."
EXAMINER'S RESPONSE: Examiner agrees. The rejections of Claims 1-7, 9-17, and 18-25 under 35 U.S.C. 101 has been withdrawn in light of arguments and/or amendments.
APPLICANT'S ARGUMENT: Applicant argues (page 11, paragraphs 2-4) that 'The Office responded to the Applicant's remarks with citations to passages in the MPEP. However, neither MPEP 2106.05(a) nor 2106.05(f) have any provision alleviating an Examiner from meeting the positive requirement that any finding regarding whether something is well-understood, routine, or conventional must be factually supported. ... ¶ The rejection of claims 9 and 11 is traversed because the rejection is based on findings that various elements of claim 1 are 'well-understood, routing and conventional activity', which is unsupported by any factual findings. ... ¶ The rejection is traversed because it lacks any of the three required written citations supporting the rejection's conclusion of various limitations being well-understood, routine, or conventional. Withdrawal of the rejection is respectfully requested."
EXAMINER'S RESPONSE: Examiner notes that applicant's argument are moot. The rejections of Claims 1-7, 9-17, and 18-25 under 35 U.S.C. 101 has been withdrawn in light of arguments and/or amendments.
Applicant's arguments, see pages 12-14, filed 13 July 2026, with respect to the rejections of Claims 1-7, 9-17, and 18-25 under 35 U.S.C. 103 have been fully considered but they are not persuasive because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant argues (page 12, paragraph 4) that "The rejection compares the operation branches of claim 1 to Yang's CE-Blocks. The CE-Blocks in Yang do not comprise convolutional layers. Rather, as is known in the art, CEBlocks, aka Channel Equilibrium Blocks, are blocks that serially follow normalization or convolution layers and equalizes the contributions of channels that it receives."
Applicant argues (page 13, paragraph 1) that "A CEBlock has a distinct function and does not include convolution layers ('CE can be stacked between the normalization layer and the Rectified units, making it flexible to be integrated into various CNN architectures', Shao, Section 6). ¶ To summarize, the rejection is traversed because it is premised on an incorrect interpretation of the Yang reference, namely, that 'the neural network structure of the CE-Block comprises two convolutional layers'. It does not. It performs a different function."
Applicant argues (page 14, paragraph 2) that "claim 1 recites channel expansion and clipping that is holistic with respect to a target network block. Channels are expanded and clipped taking into consideration the entire set of channels in the target network block (and their outputs prior to expansion). Yang discloses channel expansion at the scope of only a single operation branch and has no appreciation of, or ability to, shape channels within a target network block in a way that optimizes performance of the target network block as a whole."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are moot. Amended Claim 1 is now rejected under 35 USC 102 (a)(2) in view of Lin.
Examiner agrees that Applicant's cited reference (Shao) refers to a Channel Equilibrium (CE) block. Examiner respectfully disagrees that Shao's CE block pertains to the art (Yang) cited in the 35 U.S.C. 103 rejection of Claim 1 of the previous action.
Claim Objections
Claim 21 is objected to because of the following informalities: the limitation "pruning channels from the branches based on the importance meassures" contains the typographic error "meassures." Appropriate correction is required.
Claim Rejections - 35 USC § 101
The rejections of Claims 1-7, 9-17, and 18-25 under 35 U.S.C. 101 are withdrawn in light of arguments and/or amendments.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 9-16, and 19-25 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Lin, et al. (US 2021/0073644 A1, hereinafter "Lin").
Regarding Claim 1, Lin teaches:
A method ... that performs optimizing for the target network block ... (Lin, [0040]: "A machine learning model compression system and related methods (or processes) and computer-readable media are described herein that intelligently remove certain parameters of a machine learning model" and [0005]: "one or more channels of a single-branch or multi branch neural network can be removed, one or more branches of a multi-branch neural network can be removed, certain weights of a channel of a single-branch or multi branch neural network can be pruned (e.g., set to 0), any combination thereof, and/or other parameters") performed by a computing device comprising storage hardware storing a target network block and processing hardware (Lin, [0088]: "Process 600 is illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations"), the method comprising:
generating ... an extended network block of the target network block by increasing, in the storage hardware, a number of channels ... to a determined number of channels (Lin, Fig. 6, steps 602 "Identify A Set Of Filters In A Neural Network , Each Filter In The Set Of Filters Including A Plurality Of Weights," and 604, "Generate A Duplicate Set Of Filters, Each Filter In The Duplicate Set Of Filters Including A Copy Of A Filter In The Set Of Filters," where Lin's duplicate set reasonably suggests doubling the number of filters, and where each filter corresponds to a channel, per Fig. 7A) ... of a target operation branch in the target network block (Lin, [0048]: "The layer selection engine 304 processes the trained neural network 302 to identify which layers or branches to consider for removal. The layer selection engine 304 can select or identify more complex layers or branches (referred to as complex layers or branches) that consume more resources than the other") ..., wherein the target network block comprises a sub-network of a neural network comprising operation branches (Lin, Fig. 5A, depicting network layer 504, comprising branches 505, 507, and 509, where 504 corresponds to the instant target network block, and [0049]: "The current layer 504 includes a branch 505, a branch 507, and a branch 509. The branch 505 includes one or more 1x1 convolutional filters ... The branch 507 includes one or more 3x3 convolutional filters 514 over a number of channels .... The branch 509 includes one or more 5x5 convolutional filters 518 over a number of channels") that include the target operation branch (Lin, [0056]: "Because the number of weights of the branch 509 is greater than the weight thresh old, the layer selection engine 304 can select the branch 509 for compression"), and wherein each operation branch comprises channels having respective outputs and each operation branch is a separate computational pathway within the neural network (Lin, [0049]: "The branch 505 includes one or more 1x1 convolutional filters ... The branch 507 includes one or more 3x3 convolutional filters 514 over a number of channels .... The branch 509 includes one or more 5x5 convolutional filters 518 over a number of channels" and [0051]: "An output array of values for the convolutional filter can be produced, with each value of the output array corresponding to the number representing the filter results for that location or iteration") ... by the processing hardware (Lin, [0088]: "Process 600 is illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof");
determining ... based on at least the increased channels in each of the operation branches in the extended target network block, including a target operation branch (Lin, [0063]: "The output from the set of filters of the selected layer 310 and the output from the duplicate set of filters of the duplicate layer 308 can be provided to the loss minimization engine 312. The loss minimization engine 312 can minimize a loss function that includes an error between the output of the set of filters of the selected layer 310 and the output of the duplicate set of filters of the duplicate layer 308"), importance measures of the respective operation branches, in the extended network block (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input" and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's second term corresponds to the importance measure of an extended block," given that Lin's scaling factor corresponds to a channel importance per [0066]: "The scaling factor
γ
can also be used to represent an importance of a particular channel the scaling factor is associated with") ... by the processing hardware (Lin, [0088]: "Process 600 is illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof");
clipping ... a channel of the target operation branch in the extended network block (Lin, [0092]: "FIG. 8 is a diagram illustrating an example a compressed neural network 800, which is a compressed version of the multi-branch neural network 500 of FIG. 5A. As shown in FIG. 8, a current layer 504 of the compressed neural network 800 includes the branch 505 and the branch 507, but not the branch 509"), wherein the clipping is performed according to the importance measures of the respective operation branches including the target operation branch (Lin, [0092]: "the branch 509 was identified as a complex branch based on the branch 509 having a number of weights (78 weights) greater than the weight threshold (e.g., 35). Based on minimization of a loss function by the loss minimization engine 312 and a penalty applied by the penalty engine 314 (e.g., using Equation (3)), candidate filter determination engine 316 can remove the entire branch 509, including the 5x5 convolutional filters 518 and the 1x1 convolutional filters 520") ... by the processing hardware (Lin, [0088]: "Process 600 is illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof"); and
perform, on an input, an inference operation by the neural network comprising the clipped extended network block (Lin, [0086]: "At block 616, the process 600 includes replacing the set of filters in the neural network with the modified duplicate set of filters (e.g., B is replaced with the compressed version of B*). In some examples, the process 600 includes re-training the modified duplicate set of filters using the set of training data (and in some cases the training loss) that was used to train the neural network," where Lin's training data corresponds to the instant input, and where Lin's training reasonably suggests an inference operation per [0037]: "Once the network 100 is trained, it can be used to provide an output related to a target task during inference (during run-time after the neural network model has been trained"), the inference operation comprising providing an input of the clipped extended target network block to each of the operation branches of the clipped extended network block and forming an output of the clipped extended network block by aggregating corresponding outputs of the operation branches (Lin, [0086]: "re-training the modified duplicate set of filters using the set of training data (and in some cases the training loss) that was used to train the neural network. Re-training the neural network using the training data (and the training loss in some cases) re-tunes the network so that it can complete the target task the network was designed to perform with high accuracy," where Lin's re-tuning uses the clipped block, per [0006]: "The neural network with the compressed duplicate set of filters can be fine-tuned by re-training the neural network. In some cases, during the fine-tuning, only the parameters of the compressed duplicate set of filters will be re-trained, and the layers or branches that were not selected for compression are not re-trained"), wherein a result of the inference operation is based on the output of the clipped extended network block (Lin, [0035]: "The loss (or error) will be high for the first training images since the actual values will be much different than the predicted output," where Lin's loss is calculated based on the output during training),
wherein the determining of the importance measure of each operation branch in the extended network block comprises:
determining a weight of each operation branch and a weight of each channel of each operation branch in the extended network block (Lin, [0071]: "The original complex layer or branch B includes a set of filters, and the B * (the duplicate of B) includes a duplicate set of filters. As noted above, the duplicate set of filters in B* can include an exact copy of the original set of filters in B (e.g., including the same dimensions, the same weight values, the same scaling factors, and/or other parameters)," where Lin's filter weights correspond to the instant channel weights, and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's
λ
term corresponds to the instant branch weight) according to a relationship between the outputs, including branch outputs, and the weights, wherein each branch output of a respective operation branch depends on weights and outputs of the other operation branches (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training samples X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input. When minimizing the loss, the input x is input through only the branch or layer being compressed"); and
determining the importance measure of each operation branch in the extended network block, wherein each importance is determined based on the weight of each operation branch and the weight of each channel of each operation branch (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input" and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's second term corresponds to the importance measure of an extended block," given that Lin's scaling factor corresponds to a channel importance per [0066]: "The scaling factor
γ
can also be used to represent an importance of a particular channel the scaling factor is associated with").
Regarding Claim 14, Lin teaches:
An apparatus comprising: processing hardware (Lin, [0041]: "FIG. 3 is a block diagram illustrating an example of a machine learning model compression system 300 (also referred to as a model compression system 300). ... The components of the model compression system 300 can include and/or can be implemented using ... central processing units (CPUs)"); and storage hardware storing (Lin, [0099]: "Storage device 930 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer.... Storage device 930 can be connected to the computing device connection 905") a target network block of a neural network comprising operation branches, each operation branch comprising a block of layers and channels having respective outputs, the blocks of the respective operation branches configured to operate in parallel with respect to each other (Lin, Fig. 5A, depicting block 504 comprising layers of filters, each filter corresponding to a channel, and [0049]: "FIG. 5A is a diagram illustrating an example of a part 500 of a multi-branch neural network. ... The current layer 504 includes a branch 505, a branch 507, and a branch 509. The branch 505 includes one or more 1x1 convolutional filters 512, such a first 1x1 convolutional filter in a first channel, a second 1x1 convolutional filter in a second channel"), the storage hardware also storing instructions configured to, when executed by the processing hardware, configure the processing hardware (Lin, [0088]: "Process 600 is illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations") to:
extend the target network block by adding a number of new channels among the operation branches such that a total number of channels in each operation branch becomes a preset number of channels (Lin, Fig. 6, steps 602 "Identify A Set Of Filters In A Neural Network , Each Filter In The Set Of Filters Including A Plurality Of Weights," and 604, "Generate A Duplicate Set Of Filters, Each Filter In The Duplicate Set Of Filters Including A Copy Of A Filter In The Set Of Filters," where Lin's duplicate set reasonably suggests doubling the number of filters, and where each filter corresponds to a channel, per Fig. 7A), wherein the target network block comprises a sub-network of a neural network (Lin, Fig. 5A, depicting network layer 504, comprising branches 505, 507, and 509, where 504 corresponds to the instant target network block, and [0049]: "The current layer 504 includes a branch 505, a branch 507, and a branch 509. The branch 505 includes one or more 1x1 convolutional filters ... The branch 507 includes one or more 3x3 convolutional filters 514 over a number of channels .... The branch 509 includes one or more 5x5 convolutional filters 518 over a number of channels"), and wherein each operation branch is a separate computational pathway within the neural network (Lin, [0049]: "The branch 505 includes one or more 1x1 convolutional filters ... The branch 507 includes one or more 3x3 convolutional filters 514 over a number of channels .... The branch 509 includes one or more 5x5 convolutional filters 518 over a number of channels" and [0051]: "An output array of values for the convolutional filter can be produced, with each value of the output array corresponding to the number representing the filter results for that location or iteration");
determine, based on at least the increased channels in each of the operation branches in the extended target network block (Lin, [0063]: "The output from the set of filters of the selected layer 310 and the output from the duplicate set of filters of the duplicate layer 308 can be provided to the loss minimization engine 312. The loss minimization engine 312 can minimize a loss function that includes an error between the output of the set of filters of the selected layer 310 and the output of the duplicate set of filters of the duplicate layer 308"), an importance measure of each respective operation branch in the extended network block (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input" and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's second term corresponds to the importance measure of an extended block," given that Lin's scaling factor corresponds to a channel importance per [0066]: "The scaling factor
γ
can also be used to represent an importance of a particular channel the scaling factor is associated with");
clip channels of the operation branches in the extended network block (Lin, [0092]: "FIG. 8 is a diagram illustrating an example a compressed neural network 800, which is a compressed version of the multi-branch neural network 500 of FIG. 5A. As shown in FIG. 8, a current layer 504 of the compressed neural network 800 includes the branch 505 and the branch 507, but not the branch 509") according to the importance measures of the respective operation branches (Lin, [0092]: "the branch 509 was identified as a complex branch based on the branch 509 having a number of weights (78 weights) greater than the weight threshold (e.g., 35). Based on minimization of a loss function by the loss minimization engine 312 and a penalty applied by the penalty engine 314 (e.g., using Equation (3)), candidate filter determination engine 316 can remove the entire branch 509, including the 5x5 convolutional filters 518 and the 1x1 convolutional filters 520"); and
perform an inference operation by the neural network comprising the clipped extended network block (Lin, [0086]: "At block 616, the process 600 includes replacing the set of filters in the neural network with the modified duplicate set of filters (e.g., B is replaced with the compressed version of B*). In some examples, the process 600 includes re-training the modified duplicate set of filters using the set of training data (and in some cases the training loss) that was used to train the neural network," where Lin's training data corresponds to the instant input, and where Lin's training reasonably suggests an inference operation per [0037]: "Once the network 100 is trained, it can be used to provide an output related to a target task during inference (during run-time after the neural network model has been trained"), the inference operation comprising providing an input of the clipped extended target network block to each of the operation branches of the clipped extended network block and forming an output of the clipped extended network block by aggregating corresponding outputs of the operation branches (Lin, [0086]: "re-training the modified duplicate set of filters using the set of training data (and in some cases the training loss) that was used to train the neural network. Re-training the neural network using the training data (and the training loss in some cases) re-tunes the network so that it can complete the target task the network was designed to perform with high accuracy," where Lin's re-tuning uses the clipped block, per [0006]: "The neural network with the compressed duplicate set of filters can be fine-tuned by re-training the neural network. In some cases, during the fine-tuning, only the parameters of the compressed duplicate set of filters will be re-trained, and the layers or branches that were not selected for compression are not re-trained"), wherein a result of the inference operation is based on the output of the clipped extended network block (Lin, [0035]: "The loss (or error) will be high for the first training images since the actual values will be much different than the predicted output," where Lin's loss is calculated based on the output during training),
wherein the determining of the importance measure of each operation branch in the extended network block comprises:
determining a weight of each operation branch and a weight of each channel of each operation branch in the extended network block (Lin, [0071]: "The original complex layer or branch B includes a set of filters, and the B * (the duplicate of B) includes a duplicate set of filters. As noted above, the duplicate set of filters in B* can include an exact copy of the original set of filters in B (e.g., including the same dimensions, the same weight values, the same scaling factors, and/or other parameters)," where Lin's filter weights correspond to the instant channel weights, and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's
λ
term corresponds to the instant branch weight) according to a relationship between the outputs, including branch outputs, and the weights, wherein each branch output of a respective operation branch depends on weights and outputs of the other operation branches (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training samples X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input. When minimizing the loss, the input x is input through only the branch or layer being compressed"); and
determining the importance measure of each operation branch in the extended network block, wherein each importance measure is determined based on the weight of each operation branch and the weight of each channel of each operation branch (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input" and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's second term corresponds to the importance measure of an extended block," given that Lin's scaling factor corresponds to a channel importance per [0066]: "The scaling factor
γ
can also be used to represent an importance of a particular channel the scaling factor is associated with").
Regarding Claim 21, Lin teaches:
A method performed by a computing device comprising processing hardware and storage hardware (Lin, [0088]: "Process 600 is illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations"), the method comprising: optimizing, by the processing hardware, a neural network (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the ... loss function," where Lin's optimization of the loss function results in optimal loss by weights of the network) stored in the storage hardware (Lin, [0097]: "Computing device architecture 900 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 910. Computing device architecture 900 can copy data from memory 915 and/or the storage device 930 to cache 912 for quick access by processor 910"), the optimizing comprising:
selecting a network block from the neural network (Lin, Fig. 6, step 602, "Identify A Set Of Filters In A Neural Network , Each Filter In The Set Of Filters Including A Plurality Of Weights"), the network block comprising a sub-network of the neural network comprising branches, the branches configured to operate in parallel with respect to each other ..., wherein each branch is a separate computational pathway within the neural network (Lin, Fig. 5A, depicting layer 504 corresponding to the instant block, and branches 505, 507, and 509, each branch representing a parallel path in the block)... , each branch comprising a respective original number of original channels (Lin, [0049]: "The branch 505 includes one or more 1x1 convolutional filters ... The branch 507 includes one or more 3x3 convolutional filters 514 over a number of channels .... The branch 509 includes one or more 5x5 convolutional filters 518 over a number of channels" and [0051]: "An output array of values for the convolutional filter can be produced, with each value of the output array corresponding to the number representing the filter results for that location or iteration") and comprises a respective channel weight (Lin, Fig. 6, step 602, "Identify A Set Of Filters In A Neural Network , Each Filter In The Set Of Filters Including A Plurality Of Weights");
determining numbers of extension channels to add to the branches, respectively, of the network block based at least on a totality of original channels of the branches (Lin, Fig. 6, step 604, "Generate A Duplicate Set Of Filters, Each Filter In The Duplicate Set Of Filters Including A Copy Of A Filter In The Set Of Filters," where Lin's filters correspond to the instant channels, and where Lin's duplication is based on the number of original channels), and adding the determined numbers of extension channels to the respective branches of the network block such that the network block comprises the extension channels and the original channels, the original channels having respective outputs (Lin, Fig. 6, step 608, "Process The Input Data Using The Duplicate Set Of Filters To Generate An Output Of The Duplicate Set Of Filters," and step 610, "Minimize A Loss Function Of An Error Between The Output Of The Set Of Filters And The Output Of The Duplicate Set Of Filters And A Penalty Applied To One Or More Scaling Factors Associated With The Duplicate Set Of Filters," where Lin's processing of joint losses corresponds to the instant adding channels to the block);
determining an importance measure of each branch in the network block with the added channels by:
determining a weight of each branch and a weight of each channel of each branch (Lin, [0071]: "The original complex layer or branch B includes a set of filters, and the B * (the duplicate of B) includes a duplicate set of filters. As noted above, the duplicate set of filters in B* can include an exact copy of the original set of filters in B (e.g., including the same dimensions, the same weight values, the same scaling factors, and/or other parameters)," where Lin's filter weights correspond to the instant channel weights, and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's
λ
term corresponds to the instant branch weight) according to a relationship between the outputs, including branch outputs, and the weights, wherein each branch output of a respective branch depends on weights and outputs of the other branches (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training samples X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input. When minimizing the loss, the input x is input through only the branch or layer being compressed"), and
determining each importance measure based on the weight of each branch and the weight of each channel of each operation branch (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input" and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's second term corresponds to the importance measure of an extended block," given that Lin's scaling factor corresponds to a channel importance per [0066]: "The scaling factor
γ
can also be used to represent an importance of a particular channel the scaling factor is associated with");
pruning channels from the branches (Lin, [0092]: "FIG. 8 is a diagram illustrating an example a compressed neural network 800, which is a compressed version of the multi-branch neural network 500 of FIG. 5A. As shown in FIG. 8, a current layer 504 of the compressed neural network 800 includes the branch 505 and the branch 507, but not the branch 509") based on the importance measures of the respective branches (Lin, [0092]: "the branch 509 was identified as a complex branch based on the branch 509 having a number of weights (78 weights) greater than the weight threshold (e.g., 35). Based on minimization of a loss function by the loss minimization engine 312 and a penalty applied by the penalty engine 314 (e.g., using Equation (3)), candidate filter determination engine 316 can remove the entire branch 509, including the 5x5 convolutional filters 518 and the 1x1 convolutional filters 520"); and
performing, with the neural network having the pruned network block, an inference operation (Lin, [0086]: "At block 616, the process 600 includes replacing the set of filters in the neural network with the modified duplicate set of filters (e.g., B is replaced with the compressed version of B*). In some examples, the process 600 includes re-training the modified duplicate set of filters using the set of training data (and in some cases the training loss) that was used to train the neural network," where Lin's training data corresponds to the instant input, and where Lin's training reasonably suggests an inference operation per [0037]: "Once the network 100 is trained, it can be used to provide an output related to a target task during inference (during run-time after the neural network model has been trained"), the inference operation comprising providing an input of the pruned network block to each of the branches of the pruned network block and forming an output of the pruned network block by aggregating corresponding outputs of the branches (Lin, [0086]: "re-training the modified duplicate set of filters using the set of training data (and in some cases the training loss) that was used to train the neural network. Re-training the neural network using the training data (and the training loss in some cases) re-tunes the network so that it can complete the target task the network was designed to perform with high accuracy," where Lin's re-tuning uses the clipped block, per [0006]: "The neural network with the compressed duplicate set of filters can be fine-tuned by re-training the neural network. In some cases, during the fine-tuning, only the parameters of the compressed duplicate set of filters will be re-trained, and the layers or branches that were not selected for compression are not re-trained"), wherein a result of the inference operation is based on the output of the pruned network block (Lin, [0035]: "The loss (or error) will be high for the first training images since the actual values will be much different than the predicted output," where Lin's loss is calculated based on the output during training).
Regarding Claim 2, the rejection of Claim 1 is incorporated. Lin teaches:
generating an output of the target network block by splicing outputs of all the channels of the operation branches included in the target network block (Lin, Fig. 5, 510, "Filter Combining," depicting aggregation of the filters, and Fig. 6, step 606, "Process Input Data Using The Set Of Filters To Generate An Output Of The Set Of Filters , Wherein Weights Of The Set Of Filters Are Fixed," where Lin's generated output corresponds to the instant splicing, per the instant specification at [0062]: "splicing (e.g., aggregating or pooling)").
Regarding Claim 3, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the determined number of channels is determined to be equal to a total number of channels in the target network block (Lin, Fig. 6, step 604, "Generate a duplicate set of filters, each filter in the duplicate set of filters Including a copy of a filter in the set of filters," where Lin's each of filters corresponds to a channel per Fig. 7A).
Regarding Claim 4, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the generating of the extended network block comprises increasing the number of channels of each of the respective operation branches in the target network block to the determined number of channels (Lin, Fig. 6, step 604, "Generate a duplicate set of filters, each filter in the duplicate set of filters Including a copy of a filter in the set of filters").
Regarding Claim 5, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the channel clipping is performed such that a total number of channels remaining in the clipped extended network block is less than or equal to a total number of channels in the target network block (Lin, [0094]: "pruning of channels in the complex branch is performed using the techniques described herein to generate a compressed model. Table 1 below shows the performance comparison between the full model and the compressed model. As shown, the full model with 10.22 million parameters is compressed by a factor of five (5x) to 2.05 million parameters without affecting the performance of the network").
Claim 15 incorporates substantively all limitations of Claim 5 in apparatus form and is rejected under the same rationale.
Regarding Claim 6, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the importance measure of each respective operation branch in the extended network block is also determined based on an importance measure of each respective channel thereof prior to the extending of the target network block (Lin, [0056]: "A weight threshold can be set for each branch. If the number of weights in a branch (or layer, such as in a single-branch neural network) is greater than the weight threshold, the branch can be selected (or identified) for compression by the model compression system 300" and [0058]: "once the complex layers or branches of the trained neural network 302 are selected for compression by the layer selection engine 304, the penalty engine 314 applies a penalty to filters of the selected complex layers or branches. In some cases, the penalty can be based on a scaling factor associated with a layer or branch"), and wherein
the clipping comprises selecting a target channel for clipping based on the target channel having an importance measure that is not greater than an importance threshold (Lin, [0056]: "A weight threshold can be set for each branch. If the number of weights in a branch (or layer, such as in a single-branch neural network) is greater than the weight threshold, the branch can be selected (or identified) for compression by the model compression system 300").
Claim 16 incorporates substantively all limitations of Claim 6 in apparatus form and is rejected under the same rationale.
Regarding Claim 9, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the determining of the importance measure of each operation branch in the extended network block, based on the weight of each operation branch and the weight of each channel of each operation branch comprises:
determining an importance measure of each operation branch according to a contribution number of each operation branch, each contribution number corresponding to a count of channels in each operation branch (Lin, [0071]: "The candidate (or removable) filters in the duplicate set of filters in B* can be determined by minimizing (or optimizing) the following loss function:
L
*
B
,
B
*
=
1
N
∑
x
=
1
N
B
x
-
B
*
x
2
+
λ
∑
γ
∈
T
g
γ
Equation (3)
" and [0072]: "where ... B (x) denotes the output of the complex layer or branch B taking the training X (in the mini-batch or for the entire network) as input, B*(x) denotes the output of the duplicate layer or branch B* taking the training samples x as input" and [0073]: "The second term (
λ
∑
γ
∈
T
g
γ
) in Equation (3) represents the penalty that is added to all the filters in B*, so that more filters can be identified as candidate filters," where Lin's output B*(x) is calculated based on the weights of each operation branch and each channel, and where Lin's second term is calculated according to the count of channels itself).
Claim 19 incorporates substantively all limitations of Claim 9 in apparatus form and is rejected under the same rationale.
Regarding Claim 10, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the determining of the importance measure of each operation branch in the extended network block comprises determining an importance measure of each respective operation branch based on a relationship between a weight product of each channel of each operation branch and a weight product threshold (Lin, [0067]: "The candidate filter determination engine 316 can determine that a filter (and corresponding channel) is a candidate filter (or removable filter) if the scaling factor
γ
of the filter is less than a scaling factor threshold. ... In one illustrative example, the scaling factor threshold can be set to a percentage of the maximal scaling factor (denoted as
γ
m
a
x
) in the same batch normalization layer, and any filter with a scaling factor below the scaling factor threshold can be identified as a candidate filter for removal from the candidate set of filters").
Claim 20 incorporates substantively all limitations of Claim 10 in apparatus form and is rejected under the same rationale.
Regarding Claim 11, the rejection of Claim 1 is incorporated. Lin teaches:
selecting the target network block from the neural network (Lin, Fig. 6, step 602, "Identify a set of filters in a neural network , each filter in the set of filters Including a plurality of weights" and Fig. 5, depicting layer 504 corresponding to the instant block), which is stored in the storage hardware (Lin, [0097]: "Computing device architecture 900 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 910. Computing device architecture 900 can copy data from memory 915 and/or the storage device 930 to cache 912 for quick access by processor 910").
Regarding Claim 12, the rejection of Claim 1 is incorporated. Lin teaches:
selecting a network block from neural network, and adding an operation branch to the selected network block (Lin, Fig. 6, steps 602, "Identify A Set Of Filters In A Neural Network, Each Filter In The Set Of Filters Including A Plurality Of Weights," 604, "Generate A Duplicate Set Of Filters, Each Filter In The Duplicate Set Of Filters Including A Copy Of A Filter In The Set Of Filters," 606, "Process Input Data Using The Set Of Filters To Generate An Output Of The Set Of Filters, Wherein Weights Of The Set Of Filters Are Fixed," and 608, "Process The Input Data Using The Duplicate Set Of Filters To Generate An Output Of The Duplicate Set Of Filters," where step 602 corresponds to the instant selecting, and step 604 corresponds to the instant generating, and where the processing step 608 corresponds to adding the duplicate set to the network).
Regarding Claim 13, the rejection of Claim 1 is incorporated. Lin teaches:
wherein the target network block comprises one network block that is a sub-network of the neural network (Lin, Fig. 5A, depicting layer 504 as a network block, which comprises branches 505, 507, and 509, each of which or their combinations may be a block, per instant specification at [0067]: "an operation branch may itself correspond to a network block, and the target network block may include another network block as a part of the structure thereof"), and wherein the method further comprises:
generating a transition network block by clipping at least one operation branch in the network block according to the importance measure of each operation branch (Lin, [0091]: "the penalty engine 314 can apply a penalty to the channels of the layer 702 based on minimization of the loss by the loss minimization engine ... For example, as shown in FIG. 7B, the penalty applied by the penalty engine 314 during minimization of the loss can result in the scaling factor S2 of channel 2 ... being minimized to a value below the scaling factor threshold. The candidate filter determination engine 316 can determine that the channels 2 and 3 include candidate filters, and can remove the candidate filters (and thus the channels) from the neural network"), and generating the target network block by increasing a number of channels of at least one operation branch in the transition network block (Lin, Fig. 6, step 604, "Generate A Duplicate Set Of Filters , Each Filter In The Duplicate Set Of Filters Including A Copy Of A Filter In The Set Of Filters), wherein
a total number of channels of the target network block is less than a total number of channels in the network block (Lin, Fig. 5, depicting layer 504 comprising multiple branches, and branch 505 comprising a single branch).
Regarding Claim 22, the rejection of Claim 21 is incorporated. Lin teaches:
wherein at least one branch in the pruned network block comprises a plurality of the original channels and a plurality of the extension channels, and wherein a total number of channels in the pruned network block equals a total number of the original channels in the network block before the adding of the extension channels (Lin, [0075]: "After removing the removable filters in B* by minimizing Equation (3), the layer or branch B is replaced in the trained neural network 302 with the compressed layer or branch B*. Based on the loss minimization described above, the compressed layer or branch B* ... will have fewer parameters (e.g., channels, branches, weights, etc.)" and [0086]: "At block 614, the process 600 includes generating a modified duplicate set of filters by removing the one or more candidate filters from the duplicate set of filters. For example, after the candidate filters are removed from B* based on the minimization of the loss function, a compressed version of B* is output," where Lin's removal of weights reasonably suggests maintaining an original total number of channels, and where Lin's replacement of channels reasonably suggests maintaining an original total number of channels).
Regarding Claim 23, the rejection of Claim 21 is incorporated. Lin teaches:
selecting, from among the branches, a branch for pruning, or for pruning a channel thereof, based on the importance measures (Lin, [0065]: "the penalty engine 314 can apply a penalty to the duplicate set of filters. ... In some cases, the penalty can be based on one or more scaling factors of the duplicate set of filters. A scaling factor can be denoted by the scale parameter
γ
(e.g., as used in batch normalization layers), which can be used to detect the candidate filters" where [0066]: "The scaling factor
γ
can also be used to represent an importance of a particular channel the scaling factor is associated with").
Regarding Claim 24, the rejection of Claim 23 is incorporated. Lin teaches:
wherein the importance measure of a corresponding branch is generated based on the channel weights thereof (Lin, [0056]: "Because the number of weights of the branch 509 is greater than the weight thresh old, the layer selection engine 304 can select the branch 509 for compression," where Lin's importance measure for branch 509 is generated subsequent to identifying that 509 is a candidate for compression, and thus according to the weights).
Regarding Claim 25, the rejection of Claim 24 is incorporated. Lin teaches:
wherein a target channel is selected from among the extension and original channels of the target branch based on the selection of a target branch (Lin, Fig. 7A, depicting the pruning result of channels 3 and 4 from their branch, which is selected for consideration by inherency, where [0020]: "FIG. 7A is a diagram illustrating an example of a branch of a multi-branch neural network").
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over by Lin, et al. (US 2021/0073644 A1, hereinafter "Lin") in view of Gorokhov, et al., (US 2021/0027166 A1, hereinafter "Gorokhov").
Regarding Claim 7, the rejection of Claim 6 is incorporated. Lin teaches:
wherein the clipping is performed such that, when the total number of remaining channels in the clipped extended network block is less than a total number of channels in the target network block, the importance threshold satisfies a requirement of a ratio of a number of channels of each operation branch in the clipped extended network block to a number of channels of a corresponding operation branch in the target network block (Lin, [0056]: "A weight threshold can be set for each branch. If the number of weights in a branch (or layer, such as in a single-branch neural network) is greater than the weight threshold, the branch can be selected (or identified) for compression by the model compression system 300. In one illustrative example, referring to FIG. 5A, the weight thresh old can be set to a value of 35. Assuming the number of channels of each of the branches 505, 507, and 509 is three channels ... the branch 509 will have 78 total weights (25 weights for each of the 5x5 convolutional filters in each of the three channels, and one weight for each of the 1x1 convolutional filters in each of the three channels). The number of weights of the branch 509 (78 weights) is greater than the weight threshold of 35 and can be considered as a complex branch.... Because the number of weights of the branch 509 is greater than the weight threshold, the layer selection engine 304 can select the branch 509 for compression," where Lin's comparison of weights reasonably suggests a ratio of all of the channels of branch 509 before compression to all of the channels of branch 509 after compression, and the ratio is of weights of the branch before and after compression, and where the importance threshold satisfies the requirement that Lin's complex branches be pruned and non-complex branches not be pruned, per [0066]: "the scaling factor y can also be used to represent an importance of a particular channel the scaling factor is associated with").
Lin teaches a ratio between a target network block and a pruned extended network block.
Lin does not explicitly teach a ratio of a number of channels of each operation branch in the clipped extended network block to a number of channels of a corresponding operation branch in the target network block to be equal to or greater than 0.2 and less than or equal to 1 and, and the ratio corresponding to each operation branch satisfies a requirement that all the ratios are not 1.
However, Gorokhov teaches:
a ratio of a number of channels of each operation branch in the clipped extended network block to a number of channels of a corresponding operation branch in the target network block to be equal to or greater than 0.2 and less than or equal to 1 and, and the ratio corresponding to each operation branch satisfies a requirement that all the ratios are not 1 (Gorokhov, [0021]: "The adaptive bias component 48 may calculate a bias that controls the ratio between positive and negative values in the probability vector from the second FC layer 46, where the ratio may be set based on a fixed pruning ratio constraint (e.g., 80% important, 20% unimportant). ... Thus, passing an activation map (not shown) and the output of the threshold layer 50 through a multiplier 52 may selectively exclude all neurons in the less important channels, with importance being enforced via the pruning ratio constraint").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Lin regarding a ratio between a target network block and a pruned extended network block with those of Gorokhov regarding a ratio of a number of channels of each operation branch in the clipped extended network block to a number of channels of a corresponding operation branch in the target network block to be equal to or greater than 0.2 and less than or equal to 1 and, and the ratio corresponding to each operation branch satisfies a requirement that all the ratios are not 1.
The motivation to do so would be to facilitate processing a network to improve efficiency without sacrificing the accuracy of the network (Gorokhov, [0020]: "accuracy may be increased while accelerating inferences, regardless of the image context encountered after deployment of the neural network. For example, if the neural network is deployed in an application that processes images lacking features that were present in the images used in training, the illustrated pruning approach is able to reduce processing time by eliminating the channels configured to extract the missing features. Moreover, the technology described herein may facilitate the discard of some insignificant features that may otherwise prevent the network from making an accurate decision").
Claim 17 incorporates substantively all limitations of Claim 7 in apparatus form and is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights