DETAILED ACTION
Examiner Remarks
In light of Applicant’s Remarks submitted on 03/27/2026, Examiner has withdrawn the objections to the drawings since the outputs produced by the modules are not essential for a proper understating of the disclosed invention and the drawings do show the respective modules that execute the computations.
Response to Arguments
Applicant’s arguments with respect to claims 1, 8 and 15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3-8, 10-15, and 17-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1, 8 and 15, the phrase “based on the unified input weights, the updated pruning mask, the pruned input weights, and the unified input weights”(emphasis added) renders the claim indefinite because it is unclear why updating the hyperparameter is based on two times of the unified input weights. It is unclear why the unified input weights are recited two times in relation to updating the hyperparameter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1, 3, 5, 7-8, 10, 12, 14-15, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. "Admm-nn: An algorithm-hardware co-design framework of dnns using alternating direction methods of multipliers." Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. (2019)(“Ren”) in view of Wang et al. "Structured pruning for efficient convolutional neural networks via incremental regularization." IEEE Journal of Selected Topics in Signal Processing 14.4 (2019)(“Wang”) and in view of Han, Song, et al. "Learning both weights and connections for efficient neural network." Advances in neural information processing systems 28 (2015)(“Han”) and further in view of Hu., et al CN 110197257 A(“Hu”).
Regarding 1, Ren teaches a method of neural network model compression,
the method being performed by at least one processor(Ren, pg. 931, “Algorithm implementations are on the open-source Caffe tool with code/model release, and DNN training and compression are performed using NVIDIA Tesla P100 and GeForce GTX 1080Ti GPUs[the method being performed by at least one processor].”), and the method comprising:
receiving an input neural network and an input mask (Ren, pg. 931, “We initialize ADMM using pretrained DNN models[receiving an input neural network] and then perform weight pruning/quantization” & Ren, pg. 928, “For the weight pruning problem, the constraint set
S
i
={the number of nonzero weights is less than or equal to
α
i
}…[a]s
g
i
(
⋅
)
is the indicator function of
S
i
, the analytical solution…is
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[and an input mask].”); compressing the input neural network using a deep neural network that is trained by jointly pruning weights and unifying weights(Ren, pgs. 929-939, see also fig. 2 and fig. 3 , “Both weight pruning and quantization problems can be effectively solved using the ADMM framework… we perform weight pruning and quantization in two steps. We choose to perform weight pruning first, and then implement weight quantization on the remaining, non-zero weights[compressing the input neural network using a deep neural network that is trained by jointly]… the most important parameters in the ADMM-based weight pruning step are the
α
i
values for each layer i[pruning weights]… Fig. 3 (c) is the weights represented in quantization levels. Note that quantization levels encoded in binary bits are the operands to be stored and operated in the hardware. For the case of Fig. 3, quantization levels {−4, −3, −2, −1, 1, 2, 3, 4} are encoded using 3 binary bits, since 0 denoting pruned weights is not needed[and unifying weights].”)the jointly pruning weights and unifying weights comprising:
obtaining pruning micro-structure blocks to be pruned, from a plurality of blocks of input weights of the deep neural network [that are masked by the input mask](Ren, pg. 929, “The most important parameters in the ADMM-based weight pruning step are the
α
i
values for each layer i…[w]hen targeting high compression ratio, we reduce the
α
i
values proportionally for each layer. When targeting computation reductions, we deduct the
α
i
values for convolutional (CONV) layers[obtaining pruning micro-structure blocks to be pruned, from a plurality of blocks of input weights of the deep neural network]….”),1
dimensions of the pruning micro-structure blocks being based on an underlying hardware design of the at least one processor(Ren, pgs., 933-936, see also fig. 5, “The hardware implementations are synthesized in SMIC 40nm CMOS process using Synopsys Design Compiler[being based on an underlying hardware design of the at least one processor]...[And as fig. 5 details]
α
i
denotes denote the portion of remaining weights in layer i after weight pruning, and
1
α
i
denotes the pruning ratio in layer i. This is an iterative procedure where the amount of reduction
∆
α
i
in each iteration...[t]he underlying principle is to reduce the computation to a larger extent in those layers that are more computationally intensive[dimensions of the pruning micro-structure blocks].”);
for a first number of layers, pruning the plurality of blocks of the input weights, based on selected pruning micro-structure blocks and a pruning ratio(Ren, pg. 931, see also Table 1, “Table 1 shows the weight pruning results on the LeNet-5 model….”);
obtaining unification micro-structure blocks to be unified, from the plurality of blocks of the input weights masked by the input mask(Ren, pg. 929, “After weight pruning is performed, the next step is weight quantization on the remaining, non-zero weights [masked by the input mask]. We use n bits for equal-distance quantization to facilitate hardware implementations, which means there are a total of M =
2
n
quantization levels. More specifically, for each layer i, we quantize the weights into a set of quantization values
-
M
2
q
i
,
…
,
-
2
q
i
,
-
q
i
,
q
i
,
2
q
i
,
…
,
M
2
q
i
[obtaining unification micro-structure blocks to be unified, from the plurality of blocks of the input weights].Please note that 0 is not a quantization value because it means that the corresponding weight has been pruned[masked by the input mask].”),
dimensions of the unification micro-structure blocks being based on an underlying hardware design of the at least one processor(Ren, pgs., 933-936, see also fig. 5, “The hardware implementations are synthesized in SMIC 40nm CMOS process using Synopsys Design Compiler[being based on an underlying hardware design of the at least one processor]...[And as fig. 5 details]
α
i
denotes denote the portion of remaining weights in layer i after weight pruning, and
1
α
i
denotes the pruning ratio in layer i. This is an iterative procedure where the amount of reduction
∆
α
i
in each iteration...[t]he underlying principle is to reduce the computation to a larger extent in those layers that are more computationally intensive[dimensions of the unification micro-structure blocks].”);
for a second number of layers, unifying multiple weights in one or more of the plurality of blocks of the input weights based on the selected unification microstructure block and a unification ratio, to obtain pruned input weights and unified input weights of the deep neural network(Ren, pgs. 929-931, see also fig. 3, “[F]or each layer i, we quantize the weights into a set of quantization values
-
M
2
q
i
,
…
,
-
2
q
i
,
-
q
i
,
q
i
,
2
q
i
,
…
,
M
2
q
i
[for a second number of layers, unifying multiple weights in one or more of the plurality of blocks of the input weights based on the selected unification microstructure block]…the interval
q
i
is the distance between two adjacent quantization values, and may be different for different layers[and a unification ratio,]… Fig. 3 (a) is the weights to be quantized, obtained after pruning…Fig. 3 (c) is the weights represented in quantization levels. Note that quantization levels encoded in binary bits are the operands to be stored and operated in the hardware. For the case of Fig. 3, quantization levels {−4, −3, −2, −1, 1, 2, 3, 4} are encoded using 3 binary bits…[w]eights in quantization levels (Fig. 3 (c) ) times
q
i
= 0.5 resulting in quantized weights (Fig. 3 (b))[to obtain pruned input weights and unified input weights of the deep neural network].”);
and updating the pruned input weights and the updated input mask, based on the updated pruning mask, to minimize a loss of the deep neural network(Ren, pg. 928, “For the weight pruning problem, the constraint set
S
i
={the number of nonzero weights is less than or equal to
α
i
}…[w]e then incorporate auxiliary variables
Z
i
…
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
[to minimize a loss of the deep neural network] where
U
i
k
is the dual variable updated in each iteration
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
…}…[a]s
g
i
(
⋅
)
is the indicator function of
S
i
, the analytical solution…is
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[ and updating the pruned input weights and the updated input mask, based on the updated pruning mask]…[i]n the [above] objective function…the first term is the differentiable loss function of DNN…[t]his problem can be solved by stochastic gradient descent[to minimize a loss of the deep neural network]…”),
the loss of the deep neural network being a combination of a compression rate loss, a unification distortion loss, and a computation speed loss, wherein the computation speed loss indicates an estimated computation speed of using the unified input weights(Ren, page 928 and pages. 934-935, see also fig. 5 and Tables 8 and 9, “[t]he problem of weight pruning and quantization is an optimization problem [as detailed in equation (3)][ the loss of the deep neural network being a combination of a compression rate loss, a unification distortion loss]…[b]ased on the efficient calculation of such break-even pruning ratios, we develop efficient hardware-aware DNN model compression algorithm…we are able to reduce
α
i
values for different i…[t]he amount of reduction
∆
α
i
in each iteration is proportional to
C
i
…[t]he next step is to check whether the pruning ratios
1
α
i
surpass the hardware-specific break-even pruning ratio…the number of multiply-and-accumulation (MAC) operations…[are] directly related to…hardware performance(speed)[ and a computation speed loss, wherein the computation speed loss indicates an estimated computation speed of using the unified input weights].”);
and obtaining an output neural network with updated weight coefficients [and updated optimal weight mask based on a pruning mask associated with the output neural network] a unifying mask associated with the output neural network, the pruned input weights, and the unified input weights such that the output neural network comprises fewer parameters than the input neural network (Ren, pg. 931, “In this section we perform comparisons on the joint weight pruning and quantization results. Table 5 presents the results on LeNet-5, while Table 6 presents the results on AlexNet, VGGNet, and ResNet-50. We can simultaneously achieve 167× pruning ratio on LeNet-5, with an average of 2.78-bit for weight representation (fewer-bit representation for FC layers and more-bit for CONV layers)[ and obtaining an output neural network with updated weight coefficients a unifying mask associated with the output neural network, the pruned input weights, and the unified input weights such that the output neural network comprises fewer parameters than the input neural network].”).2
While Ren does teach selecting pruning micro-structure blocks to be pruned, from a plurality of blocks of input weights of the deep neural network, Ren does not teach: that are masked by the input mask; and updated optimal weight mask based on a pruning mask associated with the output neural network.
However, Wang teaches: that are masked by the input mask(Wang, pg. 780, see also Algorithm 1, “[F]or SpatialReg, there are H ×W weight groups. Then, we use IncReg in Algorithm 1 to prune the convolutional kernel into smaller ones…[w]e formulate the reshaping as an optimization problem, i.e., we look for the regular shape which has the minimal difference
from the irregular one, where the difference is defined as
V
(
i
)
=
∑
s
=
0
S
|
M
s
-
m
i
s
|
where S denotes the number of spatial groups…M(s) is the mask value of position s[that are masked by the input mask]”);
and updated optimal weight mask based on a pruning mask associated with the output neural network(Wang, pg. 780, see also Algorithm 1, “[F]or SpatialReg, there are H ×W weight groups. Then, we use IncReg in Algorithm 1 to prune the convolutional kernel into smaller ones[based on a pruning mask associated with the output neural network]…[w]e formulate the reshaping as an optimization problem, i.e., we look for the regular shape which has the minimal difference from the irregular one, where the difference is defined as
V
(
i
)
=
∑
s
=
0
S
|
M
s
-
m
i
s
|
…
m
(
i
)
(
s
)
is the mask value of the ith sub-retangle…[s]patial- Reg aims to find a sub-rectangle which minimizes Equation (7)[ and updated optimal weight mask].”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ren with the teachings of Wang the motivation to do so would be to treat weights of a neural network in an individual manner when it comes to improving regularization of neural networks(Wang, pg. 776, “The proposed method essentially provides a scheme to assign different regularization factors to different weights for better sparsity and performance trade-off. As shown in Fig. 2, IncReg can be readily incorporated into a typical neural network training process, which is easy to implement on various deep learning platforms.”).
While Ren in view of Wang teach the input mask Ren in view of Wang do not teach: updating the input mask and a pruning mask based on fixing unified weights in the one or more blocks of input weights and updating remaining weights in the one or more blocks of input weights, based on the selected pruning micro-structure blocks.
However, Han teaches:
updating the input mask and a pruning mask based on fixing unified weights in the one or more blocks of input weights and updating remaining weights in the one or more blocks of input weights, based on the selected pruning micro-structure blocks(Han, pgs. 3-4, “We fix the parameters for CONV layers[based on fixing unified weights in the one or more blocks of input weights] and only retrain the FC layers after pruning the FC layers, and vice versa[and updating remaining weights in the one or more blocks of input weights, based on the selected pruning micro-structure blocks]... Caffe was modified to add a mask which disregards pruned parameters during network operation for each weight tensor[updating the input mask and a pruning mask]. The pruning threshold is chosen as a quality parameter multiplied by the standard deviation of a layer’s weights.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ren in view of Wang with the teachings of Han the motivation to do so would be to prune a network that is able to preserve the original accuracy of the network(Han, pg., 2, “We present a method to prune network connections in a manner that preserves the original accuracy. After an initial training phase, we remove all connections whose weight is lower than a threshold. This pruning converts a dense, fully-connected layer to a sparse layer. This first phase learns the topology of the networks — learning which connections are important and removing the unimportant connections. We then retrain the sparse network so the remaining connections can compensate for the connections that have been removed.”).
Ren in view of Wang and Han do not teach: and for a next iteration of jointly pruning weights and unifying weights, updating a hyperparameter to increase in value based on the unified input weights, the updated pruning mask, the pruned input weights, and the unified input weights.
However, Hu teaches:
and for a next iteration of jointly pruning weights and unifying weights, updating a hyperparameter to increase in value based on the unified input weights, the updated pruning mask, the pruned input weights, and the unified input weights(Hu, pgs., 3-4, “[I]n said step (2.5) is positive then increment
∆
λ
g
(
r
)
the specific distribution mode according to the ranking r...ranking r, is different for the function independent variable parameter group distributes the corresponding regularization increment
∆
λ
g
(
r
)
wherein, A and U are hyper-parameter to be manually set...[f]urther, the specific process of said step (3) is, for the network sparsity, and then marking the parameters in the parameter group has been deleted by the mask...according to the relative importance of each weight group, gradually is different between the different regularization increment assigned to different weight group...only positive weight group then factor reaches a specified limit of regularization weight will be the weight permanent deleting...the invention for pruning, firstly all regularization factor is initialized to 0, and then gradually increasing the regularization factor by means of progressive in the iterative process.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ren in view of Wang and Han with the teachings of Hu the motivation to do so would be to incrementally regularize the network to determine the relative importance of each pruning decision (Hu, pg., 2, “At the same time, when the method for pruning, firstly all regularization factor is initialized to 0, then relative importance based on each weight group, the method of progressive updating gradually increasing the regularization factor, solving the problem that the traditional algorithm difficult to bear a larger penalty due to CNN weak expressive when pruning starts, so as to reduce the performance loss of the model.”).
Regarding claim 3, Ren in view of Wang, Han and Hu teaches the method of claim 1, wherein the deep neural network is further trained by:
reshaping the plurality of blocks of the input weights masked by the input mask(Wang, pg. 780, “Hence, for SpatialReg, there are H × W weight groups. Then, we use IncReg in Algorithm 1 to prune the convolutional kernel into smaller ones…[w]e formulate the reshaping as an optimization problem[reshaping the input weights], i.e., we look for the regular shape which has the minimal difference from the irregular one, where the difference is defined as
V
(
i
)
=
∑
s
=
0
S
|
M
s
-
m
i
s
|
where S denotes the number of spatial groups…M(s) is the mask value of position s[masked by the input mask].”);
partitioning the reshaped input weights into the plurality of blocks of the input weights(Wang, pg. 780, As fig. 5 details:
PNG
media_image1.png
138
523
media_image1.png
Greyscale
(a) is an example of reshaping large irregularly-shaped kernels into regular ones. First, the kernels are pruned by IncReg, where blue color indicates pruned positions and white color indicates unpruned ones. By using the SpatialReg algorithm, the irregular shape has been changed into a regular one[partitioning the reshaped input weights into the plurality of blocks of the input weights].);
unifying the multiple weights in one or more of the plurality of blocks(Ren, pg. 929, “After weight pruning is performed, the next step is weight quantization on the remaining, non-zero weights. We use n bits for equal-distance quantization to facilitate hardware implementations, which means there are a total of M =
2
n
quantization levels. More specifically, for each layer i, we quantize the weights into a set of quantization values
-
M
2
q
i
,
…
,
-
2
q
i
,
-
q
i
,
q
i
,
2
q
i
,
…
,
M
2
q
i
[unifying multiple weights in one or more of the plurality of blocks].”)
into which the reshaped input weights are partitioned, among the input weights(Wang, pg. 780, As fig. 5 details:
PNG
media_image1.png
138
523
media_image1.png
Greyscale
(a) is an example of reshaping large irregularly-shaped kernels into regular ones. First, the kernels are pruned by IncReg, where blue color indicates pruned positions and white color indicates unpruned ones. By using the SpatialReg algorithm, the irregular shape has been changed into a regular one[into which the reshaped input weights are partitioned, among the input weights]);
updating the input mask and a unifying mask indicating whether each of the input weights is unified, based on the unified multiple weights in the one or more of the plurality of blocks(Ren, pgs. 928-929, “For the weight pruning problem, the constraint set
S
i
={the number of nonzero weights is less than or equal to
α
i
}…[a]s
g
i
(
⋅
)
is the indicator function of
S
i
, the analytical solution…is
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[updating the input mask and a unifying mask indicating whether each of the input weights is unified, based on the unified multiple weights in the one or more of the plurality of blocks].”);
and updating the updated input mask and the input weights among which the multiple weights in the one or more of the plurality of blocks are unified (Ren, pg. 928, “As
g
i
(
⋅
)
is the indicator function of
S
i
, the analytical solution…is
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[and updating the updated input mask and the input weights among which the multiple weights in the one or more of the plurality of blocks are unified].”),
based on the updated unifying mask, to minimize the loss of the deep neural network(Ren, pg. 928, “
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[based on the updated unifying mask]. In the objective function of (5), the first term is the differentiable loss function of DNN[to minimize a loss of the deep neural network]… [t]his problem can be solved by stochastic gradient descent (e.g., ADAM)[to minimize a loss of the deep neural network]”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ren with the above teachings of Wang for the same rationale stated at Claim 1.
Regarding claim 5, Ren in view of Wang, Han and Hu teaches the method of claim 1, wherein the deep neural network is further trained by
updating a unifying mask indicating whether each of the plurality of blocks of the input weights is unified, based on the unified multiple weights in the one or more of the plurality of blocks(Ren, pg. 928, “
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[updating a unifying mask indicating whether each of the input weights is unified, based on the unified multiple weights in the one or more of the plurality of blocks].”),
wherein the updating the input mask comprises updating the input mask, based on the selected pruning micro-structure blocks and the selected unification micro-structure blocks to obtain a pruning-unification mask (Ren, pgs. 928-929, “For the weight pruning problem, the constraint set
S
i
={the number of nonzero weights is less than or equal to
α
i
…[a]s
g
i
(
⋅
)
is the indicator function of
S
i
, the analytical solution…is
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[ updating the input mask, based on the selected pruning micro-structure blocks and the selected unification micro-structure blocks to obtain a pruning-unification mask].”),
and wherein the updating the pruned input weights and the updated input mask comprises updating the pruned and unified input weights and the pruning-unification mask, based on the updated pruning mask and the updated unifying mask, to minimize the loss of the deep neural network(Ren, pgs. 928-929, “The first subproblem is…(Eqn.(5)) where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
…[t]he derived
Z
i
k
+
1
will be fed into the first subproblem in the next iteration[updating the pruned and unified input weights and the pruning-unification mask, based on the updated pruning mask and the updated unifying mask]. The intuition of ADMM is as follows. In the context of DNNs, the ADMM-based framework can be understood as a smart regularization technique. Subproblem 1 (Eqn. (5))performs DNN training with an additional L2 regularization term, and the regularization target
Z
i
k
-
U
i
k
is dynamically updated in each iteration through solving subproblem 2[Eqn. (6))][ to minimize the loss of the deep neural network].”).
Regarding claim 7, Ren in view of Wang, Han and Hu teaches the method of claim 1, wherein the pruning micro-structure blocks are selected from the plurality of blocks of the input weights masked by the input mask(Wang, pg. 780, “Hence, for SpatialReg, there are H × W weight groups. Then, we use IncReg in Algorithm 1 to prune the convolutional kernel into smaller ones[wherein the pruning micro-structure blocks are selected from the plurality of blocks of the input weights]…we look for the regular shape which has the minimal difference from the irregular one, where the difference is defined as
V
(
i
)
=
∑
s
=
0
S
|
M
s
-
m
i
s
|
where S denotes the number of spatial groups…M(s) is the mask value of position s[masked by the input mask]”), based on a predetermined pruning ratio of the input weights to be pruned for each iteration(Wang, pg. 778, “For the pruning task, we need to set a pruning ratio R to each layer, say, R = 0.6 means that we need to prune 60% weight groups when the pruning is finished[based on a predetermined pruning ratio of the input weights to be pruned for each iteration].”).
It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of Ren with the above teachings of Wang for the same rationale stated at Claim 1.
Referring to independent claim 8, Ren teaches at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising(Ren, pg. 933, “Using the proposed ADMM framework, the total model size of VGGNet is reduced to 8.3MB (using 6.9M weights) when the indices are accounted for. This model size can still be accommodated by a single high-end FPGA such as Altera (Intel) DE-5 and Xilinx Virtex-7[and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising]. The effect that large-scale AlexNet and VGGNet models can be stored using on-chip memory of single FPGA/ASIC[at least one memory configured to store program code]….”) and for all other claim limitations claim 8 is rejected on the same basis as independent claim 1 since they are analogous claims.
Referring to dependent claims 10, 12 and 14, they are rejected on the same basis as
dependent claims 3, 5, and 7 since they are analogous claims
Referring to independent claim 15, storing instructions that, when executed by at least one processor for neural network model compression, cause the at least one processor to(Ren, pg. 933, “Using the proposed ADMM framework, the total model size of VGGNet is reduced to 8.3MB (using 6.9M weights) when the indices are accounted for. This model size can still be accommodated by a single high-end FPGA such as Altera (Intel) DE-5 and Xilinx Virtex-7[storing instructions that, when executed by at least one processor for neural network model compression, cause the at least one processor to]. The effect that large-scale AlexNet and VGGNet models can be stored using on-chip memory of single FPGA/ASIC….”) and for all other claim limitations claim 15 is rejected on the same basis as independent claim 1 since they are analogous claims.
Referring to dependent claims 17 and 19 they are rejected on the same basis as
dependent claims 3, and 5 since they are analogous claims
Claims 4, 6, 11, 13, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. "Admm-nn: An algorithm-hardware co-design framework of dnns using alternating direction methods of multipliers." Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. (2019)(“Ren”) in view of Wang et al. "Structured pruning for efficient convolutional neural networks via incremental regularization." IEEE Journal of Selected Topics in Signal Processing 14.4 (2019)(“Wang”) and in view of Han, Song, et al. "Learning both weights and connections for efficient neural network." Advances in neural information processing systems 28 (2015)(“Han”) and in view of Hu., et al CN 110197257 A(“Hu”) and further in view of Lemaire et al.,"Structured pruning of neural networks with budget-aware regularization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2019)(“Lemaire”).
Regarding claim 4, Ren in view of Wang, Han and Hu teaches the method of claim 3, wherein the updating of the updated input mask and the input weights comprises:
…using the deep neural network of which the input weights are unified and masked by the updated input mask(Ren, pgs. 928-929, “The first subproblem is
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[using the deep neural network of which the input weights are unified and masked]… the analytical solution…is
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[by the updated input mask].”);
determining the loss of the deep neural network…(Ren, pg. 928, “In the objective function of (5), the first term is the differentiable loss function of DNN, and the second quadratic term is differentiable and convex…[t]his problem can be solved by stochastic gradient descent (e.g., ADAM)[ determining the loss of the deep neural network]”)
determining a gradient of the determined loss, based on the input weights among which the multiple weights in the one or more of the plurality of blocks are unified(Ren, pgs. 928-929, “The first subproblem is
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[based on the input weights among which the multiple weights in the one or more of the plurality of blocks are unified] In the objective function of (5), the first term is the differentiable loss function of DNN, and the second quadratic term is differentiable and convex…[t]his problem can be solved by stochastic gradient descent (e.g., ADAM)[ determining a gradient of the determined loss].”);
and updating the pruned input weights and the updated input mask, based on the determined gradient and the updated unifying mask, to minimize the determined loss(Ren, pg. 928-929, “The first subproblem is
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
[to minimize the determined loss]where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[and the updated unifiying mask]…
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[and updating the pruned input weights and the updated input mask]where
∏
S
i
(
⋅
)
is Euclidean projection of
W
i
k
+
1
+
U
i
k
onto the set
S
i
…[f]or both weight pruning and quantization problems, the first subproblem has the same form when
Z
i
k
is determined…[a]s a result they can be solved in the same way by stochastic gradient descent (e.g., the ADAM algorithm)[ based on the determined gradient]”).
Ren in view of Wang, Han and Hu do not teach: reducing parameters of a first training neural network, to estimate a second training neural network, based on the estimated second training neural network and a ground-truth neural network;
However, Lemaire teaches:
reducing parameters of a first training neural network, to estimate a second training neural network(Lemaire, pg. 9112, see also Algorithm 1, “Knowledge Distillation (KD)… is a method for facilitating the training of a small neural network (the student)[ reducing parameters of a first training neural network] by having it reproduce the output of a larger network (the teacher)[ to estimate a second training neural network].”),
based on the estimated second training neural network and a ground-truth neural network(Lemaire, pg. 9112, see also Algorithm 1, “The loss proposed…is:
L
D
(
W
)
=(1-
α
)
L
C
E
P
s
,
Y
g
t
+
α
T
2
L
C
E
(
P
s
,
P
t
)
where
L
C
E
is a cross-entropy,
Y
g
t
is the ground truth[and a ground-truth neural network]…
P
t
are the output logits of the…teacher networks[based on the estimated second training neural network]….”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ren in view of Wang, Han and Hu with the teachings of Lemaire the motivation to do so would be to incorporate knowledge distillation with a sparsity budget constraint to allow CNNs to be executed on various hardware devices(Lemaire, pg. 9108, “[M]ost SL methods cannot prune a network while respecting a neuron budget imposed by the very nature of a device on which the network shall be deployed. In this paper, we present a SL framework which allows learning and selecting filters of a CNN while respecting a neuron budget. Our main contributions are… [w]e demonstrate the effectiveness of combining SL and knowledge distillation.”).
Regarding claim 6, Ren in view of Wang, Han and Hu teaches the method of claim 5, wherein the updating of the pruned and unified input weights and the pruning-unification mask comprises:
…using the deep neural network of which the pruned and unified input weights are masked by the pruning-unification mask(Ren, pgs. 928-929, “The first subproblem is
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
[using the deep neural network of which the pruned and unified input weights]where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[are masked by the pruning-unification mask].”);
determining the loss of the deep neural network…(Ren, pg. 928, “In the objective function of (5), the first term is the differentiable loss function of DNN, and the second quadratic term is differentiable and convex…[t]his problem can be solved by stochastic gradient descent (e.g., ADAM)[ determining the loss of the deep neural network].”)
determining a gradient of the determined loss, based on the input weights among which the multiple weights in the one or more of the plurality of blocks are unified(Ren, pgs. 928-929, “The first subproblem is
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
[based on the input weights]where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[among which the multiple weights in the one or more of the plurality of blocks are unified]. In the objective function of (5), the first term is the differentiable loss function of DNN, and the second quadratic term is differentiable and convex…[t]his problem can be solved by stochastic gradient descent (e.g., ADAM)[ determining a gradient of the determined loss].”);
and updating the pruned and unified input weights and the pruning-unification mask, based on the determined gradient and the updated pruning mask and unifiying mask, to minimize the determined loss(Ren, pg. 928-929, “The first subproblem is
m
i
n
i
m
i
z
e
W
i
,
b
i
f
W
i
i
=
1
N
,
b
i
i
=
1
N
+
∑
i
=
1
N
ρ
i
2
W
i
-
Z
i
k
+
U
i
k
F
2
[to minimize the determined loss] where
U
i
k
is the dual variable updated in each iteration,
U
i
k
≔
U
i
k
-
1
+
W
i
k
-
Z
i
k
[and updating the pruned and unified input weights and the pruning-unification mask]…
Z
i
k
+
1
=
∏
S
i
(
W
i
k
+
1
+
U
i
k
)
[ and the updated pruning mask and unifiying mask]…[f]or both weight pruning and quantization problems, the first subproblem has the same form when
Z
i
k
is determined…[a]s a result they can be solved in the same way by stochastic gradient descent (e.g., the ADAM algorithm)[ based on the determined gradient].”).
Ren in view of Wang, Han and Hu do not teach: reducing parameters of a first training neural network, to estimate a second training neural network, based on the estimated second training neural network and a ground-truth neural network;
However, Lemaire teaches:
reducing parameters of a first training neural network, to estimate a second training neural network(Lemaire, pg. 9112, see also Algorithm 1, “Knowledge Distillation (KD)… is a method for facilitating the training of a small neural network (the student) [reducing parameters of a first training neural network]by having it reproduce the output of a larger network (the teacher)[ to estimate a second training neural network].”),
based on the estimated second training neural network and a ground-truth neural network(Lemaire, pg. 9112, see also Algorithm 1, “The loss proposed…is:
L
D
(
W
)
=(1-
α
)
L
C
E
P
s
,
Y
g
t
+
α
T
2
L
C
E
(
P
s
,
P
t
)
where
L
C
E
is a cross-entropy,
Y
g
t
is the ground truth[and a ground-truth neural network]…
P
t
are the output logits of the…teacher networks[based on the estimated second training neural network]….”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ren in view of Wang, Han and Hu with the teachings of Lemaire the motivation to do so would be to incorporate knowledge distillation with a sparsity budget constraint to allow CNNs to be executed on various hardware devices(Lemaire, pg. 9108, “[M]ost SL methods cannot prune a network while respecting a neuron budget imposed by the very nature of a device on which the network shall be deployed. In this paper, we present a SL framework which allows learning and selecting filters of a CNN while respecting a neuron budget. Our main contributions are… [w]e demonstrate the effectiveness of combining SL and knowledge distillation.”).
Referring to dependent claims11 and 13, they are rejected on the same basis as
dependent claims 4 and 6 since they are analogous claims.
Referring to dependent claims 18 and 20 they are rejected on the same basis as
dependent claims 4 and 6 since they are analogous claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADAM C STANDKE whose telephone number is (571)270-1806. The examiner can normally be reached Gen. M-F 9-9PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Adam C Standke/
Primary Examiner
Art Unit 2129
1 Examiner Remarks: The claim elements that are non-bolded and contained within brackets are not taught by the prior art of Ren
2 Examiner Remarks: The claim elements that are non-bolded and contained within brackets are not taught by the prior art of Ren
Prosecution Insights