DETAILED ACTION
The action is in response to the RCE amendment filed on 11/24/2025. Claims 1-20 are pending and have been considered below. Claims 1, 14 and 17 are independent claims. Claims 1, 14 and 17 are amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claim 1 recites “the pruning comprises removing the weights below a discontinuous change in a slope of an ordered distribution of absolute weights versus channel numbers of filters in layers…”. First of all, the specification supports pruning or removing channels not weights below a hinge or inflection point in a weight versus channel distribution graph, where the x axis indicates the channel number and the y axis indicates the absolute weights of the corresponding channel(63-64, 59). Additionally, the specification supports the pruning is done based on the channels that are below the hinge in the graph. However, the specification does not support the pruning or removing weights is based on a discontinuous change in slope of the distribution graph. There is no mention of a discontinuity in the change in slope in the specification.
Claim 14 is directed towards a non-transitory computer-readable storage medium for implementing the system of claim 1, and is likewise deficient.
Claim 17 is directed towards a method for implementing the system of claim 1, and is likewise deficient.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Claim 1 recites “the pruning comprises removing the weights below a discontinuous change in a slope of an ordered distribution of absolute weights versus channel numbers of filters in layers…”. First of all, the specification discloses pruning or removing channels not weights below a hinge or inflection point in a weight versus channel distribution graph, where the x axis indicates the channel number and the y axis indicates the absolute weights of the corresponding channel(63-64, 59). Additionally, the specification supports the pruning is done based on the channels that are below the hinge in the graph. However, the specification does not enable PHOSITA the pruning or removing weights is based on a discontinuous change in slope of the distribution graph. There is no mention of a discontinuity in the change in slope in the specification to enable this limitation.
Claim 14 is directed towards a non-transitory computer-readable storage medium for implementing the system of claim 1, and is likewise deficient.
Claim 17 is directed towards a method for implementing the system of claim 1, and is likewise deficient.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 14, 15, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (“ALF: Efficient Deep Learning-Based Lossy Image Compression via Asymmetric Autoencoder and Pruning,” hereinafter Kim) in view of Park et al. (“Prune Your Model Before Distill It,” hereinafter Park), and further in view of Park et al (hereinafter Park2) US PGPUB 20210383190 A1.
In light of the 35 USC 112a rejection above, the following rejections are set forth:
Regarding claim 1, Kim teaches a computer system, comprising: a computation device; memory configured to store program instructions (Kim, 1. Introduction, pp. 2063, col. 2, paragraph 1; “develop efficient learning-based decoders that require relatively low storage and/or computational complexity,” wherein “decoders” are necessarily comprised of program instructions and “storage” encompasses memory.) wherein, when executed by the computation device, the program instructions cause the computer system to perform operations comprising (Kim, 3.3. Effectiveness of fast decoder, pp. 2065-2066; “To evaluate the performance of our fast decoder, we measure not only the number of trainable parameters but also the number of floating point operations (FLOPs) required for decompression,” wherein the “measure [of]…FLOPs required” indicates that when executed by the computation device, the program instructions encompassing the “fast decoder” cause the electronic device to perform operations.)
obtaining information specifying an initial autoencoder (AE) neural network; (Kim, Fig. 1; The figure illustrates the initial autoencoder architecture.
Kim, 2. Efficient Lossy Image Compression, pp. 2063, col. 2, paragraph 3; “During training, instead of directly quantizing 𝑭𝐶𝐸, its continuous relaxation is performed by using additive uniform noise,” thereby training the initial autoencoder (AE) neural network. Note that obtaining information specifying an initial autoencoder (AE) neural network encompasses “training” according to the example embodiment given at paragraph [045] of the specification, “Note that obtaining the initial AE neural network may include: accessing the information specifying the initial AE neural network stored in memory associated with the computer system; training the initial AE neural network; or receiving, from another computer system, the information specifying the initial AE neural network” (emphasis added).)
computing a subset of filters associated with the initial AE neural network to remove based at least in part on a L1-norm loss function and weights associated with
filters in initial AE neural network; (Kim, 2.2 Efficient decoders via pruning, pp. 2064, col. 2, paragraph 3; “Similarly to [10, 11], we choose the element-wise ℓ1-norm of each filter (i.e., the sum of magnitudes of filter weights) as the pruning criterion. The number of filters to be pruned in each layer is determined empirically based on sensitivity analysis. After some filters are removed, the network is retrained,” wherein “choos[ing] the element-wise ℓ1-norm of each filter (i.e., the sum of magnitudes of filter weights) as the pruning criterion” denotes computing a subset of filters…to remove based at least in part on a L1-norm loss function and weights associated with filters in initial AE neural network. Specifically, “some filters [that] are removed” denotes a subset of filters.)
pruning the subset of the filters from the initial AE neural network; and (Kim, 2.2 Efficient decoders via pruning, pp. 2064, col. 2, paragraph 3; “Similarly to [10, 11], we choose the element-wise ℓ1-norm of each filter (i.e., the sum of magnitudes of filter weights) as the
pruning criterion. The number of filters to be pruned in each layer is determined empirically based on sensitivity analysis. After some filters are removed, the network is retrained,” wherein removing filters based on a “pruning criterion” encompasses pruning the subset of filters.) wherein the pruning removes incoming and outgoing connections and weights corresponding to the subset of the filters across all layers in the initial AE neural network, and wherein the pruning comprises removing the weights below a discontinuous change in a slope of (Kim, 2.2 Efficient decoders via pruning, pp. 2064, col. 2, paragraph 2; “First, we measure the pruning sensitivity of each layer to determine layer-wise pruning thresholds. Then, we remove weights whose magnitudes are smaller than the thresholds,” wherein “measur[ing] the pruning sensitivity of each layer” for the purpose of remov[ing]…weights and filters is equivalent to remov[ing] incoming and outgoing connections and weights corresponding to the subset of the filters across all layers in the initial AE neural network (emphasis added). Note that weights define the strength of connections between layers of a neural network. That the pruning of Kim takes place at “each layer” indicates that both incoming and outgoing connections are removed.)
generating a second AE neural network by retraining the initial AE neural network (Kim, 2.2. Efficient decoders via pruning, pp. 2064, col. 2, paragraph 3; “After some filters are removed, the network is retrained,” wherein the “retrained” network constitutes a second AE neural network.)
Kim does not explicitly teach wherein the retraining comprises a student-teacher model in which the teacher comprises the pruned initial AE neural network and the student comprises the second AE neural network. However, Park, in the area of model pruning and knowledge distillation, teaches this limitation (Park, Fig. 1; “Overview of the “prune, then distill” strategy. Instead of distilling directly from the teacher to the student (blue dotted box), we prune the teacher first, then distill from the pruned teacher to the student (red dotted box),” wherein “the pruned teacher” corresponds to the pruned initial AE neural network and “the student” corresponds to the second AE neural network.).
Park is analogous to the claimed invention as it is from the same field of endeavor, that is, model pruning and transfer learning. Kim teaches a method of retraining an autoencoder but does not explicitly state that this retraining constitutes a transfer learning procedure between a pruned teacher and a generated student. Park teaches this limitation. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to combine the filter pruning and retraining methods of Kim with the knowledge distillation step of Park. The motivation to do so is that pruning the teacher prior to transferring knowledge to the student offers smoother regularization and results in a better final accuracy that transferring prior to pruning (Park, Abstract; “We provide several exploratory examples where the pruned teacher teaches better than the original unpruned networks. We further show theoretically that the pruned teacher plays the role of regularizer in distillation, which reduces the generalization error.” Park, 1 Introduction, pp. 2, paragraph 3; “Knowledge distillation can be viewed as a label smoothing
regularization (LSR) [55,59], which regularizes training by providing a smoother label. We find that a teacher trained with regularization provides a smoother label than the original teacher.” Park, 3.1 Exploratory Experiments, pp. 5, paragraph 4; “Surprisingly, as shown in Table 1, VGG11 with pruned VGG19 consistently outperforms the one with the unpruned teacher. Table 1 also provides results when the teacher network is VGG19DBL, with 2 × many channels in each layer. In both cases, the pruned teacher shows better performance.” Park, Table 1).
Kim further teaches wherein the second AE neural network comprises fewer parameters than the initial AE neural network, (Kim, 2.2 Efficient decoders via pruning, pp. 2064, col. 2, paragraph 2; “First, we measure the pruning sensitivity of each layer to determine layer-wise pruning thresholds. Then, we remove weights whose magnitudes are smaller than the thresholds,” wherein to “remove weights” in creating the second “lightweight” AE neural network is equivalent to generating a network compris[ing] fewer parameters than the initial AE neural network.) executing the second AE neural network comprises reduced runtime memory, and less compute operations than executing the initial AE neural network, and (Kim, 2.2 Efficient decoders via pruning, pp. 2064, cols. 1 and 2; “On the other hand, the latter removes whole selected filters. It has the advantage that the degree of pruning directly leads to the reduction of computational complexity. In this regard, we propose lightweight and fast
decoders using weight and filter pruning, respectively,” wherein “reduction of computational complexity” is equivalent to less compute operations. This yields a second “lightweight and fast” network that necessarily comprises reduced runtime memory. Kim, Table 2;
“Performance of the proposed fast decoder (“Pruned (proposed)”) compared to the one before pruning (“Asymmetric”) and the one pruned using the method in [10] (“Pruned ([10])”) on Kodak PhotoCD [13] and the comic dataset. The columns “# params. ↓” and “FLOPs ↓” show the ratios of the reduced number of weight parameters and FLOPs, respectively,” thereby further indicating that the “Pruned” second AE neural network comprises fewer parameters, has a reduced runtime (“Rate”) and takes less compute operations (“FLOPS”) to execute.) the
second AE neural network provides improved perceptual quality for photo-realistic image
generation with image loss less than a predefined amount or zero relative to the initial AE neural network (Kim, 3.2 Effectiveness of lightweight decoder, pp. 2065, col. 2, paragraph 2; “The percentages of the number of pruned parameters range from 50.4% to 69.0%, showing that the decoder can be made significantly lightweight. It should be noted that each pruned model achieves similar or even better rate-distortion performance than the one before pruning,” wherein “an even better rate-distortion performance than the one before pruning” indicates that the second AE neural network provides improved perceptual quality for photo-realistic image
generation. As the quality is improved, the image loss is necessarily less than…zero relative to the initial AE neural network.).
Furthermore, Kim fails to explicitly teach an ordered distribution of absolute weights versus channel numbers of filters in layers in the initial AE neural network; . However, Park2 teaches eliminating at least one channel based a sum of weight values (32, 31, 3), where basically the channel numbers with the smallest sum are removed from consideration-- an ordered distribution of absolute weights versus channel numbers of filters in layers. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to combine Kim, Park, and Park2. The motivation to do so is to efficiently use a compressed neural network in various fields (10).
Regarding claim 2, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim further teaches wherein obtaining the initial AE neural network may
include: accessing the information specifying the initial AE neural network stored in memory associated with the computer system; training the initial AE neural network; or receiving, from another computer system, the information specifying the initial AE neural network (Kim, 2. Efficient Lossy Image Compression, pp. 2063, col. 2, paragraph 3; “The overall architecture of our proposed networks consists of three parts, i.e., CE [convolutional encoder], entropy coder (EC), and CD [convolutional decoder] (Fig. 1)…During training, instead of directly quantizing 𝑭𝐶𝐸, its continuous relaxation is performed by using additive uniform noise,” wherein training element “CE” of the autoencoder network denotes training the initial AE neural network.).
Regarding claim 3, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim further teaches wherein the initial AE neural network is configured to: transform an input image to a latent space, and from the latent space back to an output image (Kim, 1. Introduction, pp. 2063, col. 1, paragraph 2; “These deep learning-based methods are typically based on convolutional autoencoders. During encoding, the convolutional encoder (CE) transforms a given image into latent representation, which is compressed into a bitstream
via quantization and entropy coding. At the decoder side, the bitstream is converted to the latent representation and the convolutional decoder (CD) reconstructs the image using it.” Kim, Fig. 1).
Regarding claim 4, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim further teaches wherein the subset of filters associated with the initial AE neural network to remove are not activated or have a subset of the weights less than a predefined value (Kim, 2.2 Efficient decoders via pruning, pp. 2064, col. 2, paragraph 2; “Inspired by [9],
the pruning process is performed through three steps. First, we measure the pruning sensitivity of each layer to determine layer-wise pruning thresholds. Then, we remove weights whose magnitudes are smaller than the thresholds,” wherein “weights whose magnitudes are smaller than the thresholds” is equivalent to the weights less than a predefined value.).
Regarding claim 14, Kim teaches a non-transitory computer-readable storage medium for use in conjunction with a computer system, the computer-readable storage medium configured to store program instructions that, (Kim, 1. Introduction, pp. 2063, col. 2, paragraph 1; “develop efficient learning-based decoders that require relatively low storage and/or computational complexity,” wherein “decoders” are necessarily comprised of program instructions and “storage” encompasses a non-transitory computer-readable storage medium for use in conjunction with a computer system.) when executed by the computer system, causes the computer system to perform operations comprising (Kim, 3.3. Effectiveness of fast decoder, pp. 2065-2066; “To evaluate the performance of our fast decoder, we measure not only the number of trainable parameters but also the number of floating point operations
(FLOPs) required for decompression,” wherein the “measure [of]…FLOPs required” indicates that the program instructions when executed encompassing the “fast decoder” cause the computer system to perform operations.).
The following limitations correspond to the steps of claim 1 and are thus rejected for the same reasons as claim 1.
Claim 15 is a non-transitory computer-readable storage medium claim corresponding to the steps of claim 4 and is thus rejected for the same reasons as claim 4.
Claim 17 is a method claim corresponding to the steps claim 1 and is thus rejected for the same reasons as claim 1.
Claim 18 is a method claim corresponding to the steps of claim 4 and is thus rejected for the same reasons as claim 4.
Claims 8-10, 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Park in further view of Zhao et al. (“Enabling Deep Learning on Edge Devices through Filter Pruning and Knowledge Transfer,” hereinafter Zhao).
Regarding claim 8, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim does not explicitly teach wherein the computation is based at least in part on a type of compute environment in which the second AE neural network is intended to execute. However, Zhao, in the area of filter pruning and knowledge transfer teaches this limitation (Zhao, Abstract; “There is an increasing need of training such models on the devices in order to deliver personalized, responsive, and private learning. To address this need, this paper
presents a new solution for deploying and training state-of-the-art models on the resource-constrained devices. First, the paper proposes a novel filter-pruning-based model compression
method to create lightweight trainable models from large models trained in the cloud, without
much loss of accuracy,” wherein the “lightweight trainable models” encompassing the second AE neural network are intended to execute on smaller compute environment[s], or “resource-constrained devices.” Therefore, the [pruning] computation is done to generate a reduced but optimized second AE neural network model that can run with less computational power.).
Zhao is analogous to the claimed invention as both are from the same field of endeavor, that is, filter pruning methods. Kim teaches a method of creating a reduced second AE neural network but does not explicitly specify a compute environment that autoencoder was intended to execute on. Zhao teaches this limitation with convolutional neural networks. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to modify the pruning method of Kim to allow for specific customization of an initial model to fit a given system or environment, as taught by Zhao. The motivation to do so is to allow for deployment of flexible and scalable pruned models in a federated-like machine learning system (Zhao, 1. Introduction, pp. 1, col. 2, paragraph 1; “However, a serious drawback of this cloud-only approach is that the on-device tasks cannot perform well when the cloud is overloaded or the network is unreliable. Moreover, there are also significant benefits from training deep learning models on edge devices: 1) Customization: user- or situation-specific requirements can be met more effectively by training models on the devices that the users or physical environments directly interact with; 2) Responsiveness: custom models deployed on devices for specific users
or environments can better adapt to their changing behaviors using new data captured by the devices; and 3) Privacy: sensitive information can be better protected if the sensitive data and models are stored and used only on private devices, not in the public resources shared by many.”).
Regarding claim 9, the combination of Kim, Park and Zhao teaches the computer system of claim 8 (and thus the rejection of claim 8 is incorporated).
Kim does not explicitly teach wherein the type of compute environment comprises:
one or more processors, one or more GPUs, or both. However, Zhao, in the area of filter pruning and knowledge transfer teaches this limitation (Zhao, 5. Evaluation, pp. 5, col. 2, paragraph 3; “We implemented our solution on TensorFlow version r1.3, and evaluated the cloud model on a Nvidia Tesla K40 GPU, hosted on a server equipped with dual Intel Xeon E5-2630 processors and 64GB of main memory. We evaluate our edge model on a commercialized device, Google Pixel 2, which has an eight-core, Qualcomm Kryo 280 CPU and 4GB of main memory,” wherein the “Google Pixel 2” corresponds to the compute environment with one or more processors and one or more GPUs.).
Zhao is analogous to the claimed invention as both are from the same field of endeavor, that is, filter pruning methods. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the filter pruning method of Kim on the hardware of Zhao. The motivation to do so is inherent as an autoencoder (AE) neural network necessarily comprises instructions that require a processor for execution.
Regarding claim 10, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim does not explicitly teach wherein the initial AE neural network and the second
AE neural network are trained using a common dataset. However, Zhao, in the area of filter pruning and knowledge transfer teaches this limitation (Zhao, 5.1. Results for Model Compression, pp. 6, col. 1, paragraph 3; “We first experiment on CIFAR-10 dataset with ResNet
(WRN-28-10) as the on-server model. By changing the pruning threshold P, our method can flexibly generate four compressed models, ResNet 1-4, offering different tradeoffs between size and accuracy, shown in Table 1.” As can be seen in the table, the initial…neural network and the second…neural network (as well as subsequent pruned models) are all trained using a common dataset in each of the three experiments.).
Zhao is analogous to the claimed invention as both are from the same field of endeavor, that is, filter pruning methods. Kim teaches an initial AE neural network trained on a dataset but does not explicitly reference training an additional autoencoder on the same dataset. Zhao teaches such a method using convolutional neural networks. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to modify the combined filter pruning and knowledge transfer method of Kim and Park to train the initial and resultant models on the same dataset, as taught by Zhao. The motivation to do so is to provide accurate metric comparisons between the two models that allow for proper evaluation of the pruning methods (5.1. Results for Model Compression, pp. 6, col. 1, paragraph 3; “We first experiment on CIFAR-10 dataset with ResNet (WRN-28-10) as the on-server model. By changing the pruning threshold P, our method can flexibly generate four compressed models, ResNet 1-4, offering different tradeoffs between size and accuracy, shown in Table 1. The results show that all the compressed models can achieve good compression ratios without losing much accuracy. In particular, compressed ResNet 4, the size of which is only 0.64% of the origin model WRN-28-10, still remains a Top-1 accuracy of over 90%. ResNet 7 achieves a compression ratio of 4X at the cost of 7.16% loss in accuracy. The compressed model VGG-16 4 achieves a compression ratio of up to 139X at the cost of less than only 10% loss in accuracy.” These evaluations would be impossible or otherwise inaccurate if the models were trained on different datasets as the accuracy and performance measurements could be influenced by factors other than the pruning methods such as differences in image quality, image classes, aspect ratio, etc.).
Regarding claim 13, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim does not explicitly teach wherein a number of non-zero weights in the second AE
neural network is at least a factor of 10 less than a number of non-zero weights in the initial AE neural network. However, Zhao, in the area of filter pruning and knowledge transfer teaches this limitation (Zhao, Table 1; This table includes a column representing the number of
parameters in each of the baselines and their corresponding pruned models. The number of parameters, or non-zero weights, in the pruned models (or second models) trained on the “CIFAR-10” and “Cal-tech 101” is at least a factor of 10 less than a number of non-zero weights in each baseline, or initial, model. We know the remaining weights in the pruned model are all non-zero as the pruning threshold used by Zhao is “set between 0.7 and 1.0.”).
Zhao is analogous to the claimed invention as both are from the same field of endeavor, that is, filter pruning methods. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to generate a second model with 10 times fewer parameters than the initial model, as taught by Zhao. The motivation to do so to generate a more efficient model that can be trained using substantially fewer resources far more quickly than its cumbersome baseline model (Zhao, Table 6).
Claim 20 is a method claim corresponding to the steps of claim 8 and is thus rejected for the same reasons as claim 8.
Claims 5-7, 16 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Park in further view of Frickenstein et al. (“ALF: Autoencoder-based Low-rank Filter-sharing for Efficient Convolutional Neural Networks,” hereinafter Frickenstein).
Regarding claim 5, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim does not explicitly teach wherein the computation comprises regularizing the initial AE neural network to drive a subset of the weights associated with the subset of filters below a predefined value. However, Frickenstein, in the area of autoencoder-based pruning methods, teaches this limitation (Frickenstein, III. Autoencoder-Based Low-Rank Filter-Sharing, pp. 3, col. 2, paragraph 3; “By exploiting the sparsity-inducing property of L1 regularization, individual values in the mask M are driven towards zero during training. Since the optimizer usually reaches values close to zero, but not exactly zero, clipping is performed to zero out values that are below a certain threshold t,” wherein “individual [filter] values in the mask M are driven towards zero” is equivalent to driv[ing] a subset of weights associated with the subset of filters below a predefined value, which is “a certain threshold t.”).
Frickenstein is analogous to the claimed invention as both are directed to filter pruning for computational efficiency in neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the filter pruning method of Kim to reduce and zero out the weights of irrelevant filters, as taught by Frickenstein. The motivation to do so is to have a parameter with which one can customize and control the performance-accuracy balance of a given compressed neural network to meet the requirements of a given system (Frickenstein, IV. Experimental Results, A. Configuration Space Exploration, Setup 3, pp. 5, col. 1, paragraph 1; “Five variants of ALF, resulting in different
sparsity rates, are explored in Fig 2c and compared to the uncompressed Plain-20 (90.5% accuracy). The first three variants differ in terms of the threshold 𝑡∈ {5∙10−5,1∙10−4,5∙10−4} while the learning rate 𝑙𝑟𝑎𝑒=1∙10−3. We observe that the pruning gets more aggressive when the threshold t is increased. The number of non-zero filters remaining are 40.17%, 38.6% and 35.71% respectively (see green, pink and blue curves). We select the threshold 𝑡=1𝑒−4 as a trade-off choice between sparsity and accuracy.” Note that claims 4 and claim 5 both utilize a predefined value to control pruning of the weights. In claim 4 the filter weights are already below this value, and in claim 5 the filter weights are driven below this value. Thus, the motivation to use such a parameter remains the same in both instances: to control the level of pruning consistent with a desired performance-accuracy balance.).
Regarding claim 6, the combination of Kim and Park teaches the computer system of claim 5 (and thus the rejection of claim 5 is incorporated).
Kim does not explicitly teach wherein the regularizing is based at least in part on a number of filters in a given layer of the initial AE neural network. However, Frickenstein, in the area of autoencoder-based pruning methods, teaches this limitation (Frickenstein, III. Autoencoder-Based Low-Rank Filter-Sharing, pp. 2, col. 2, paragraph 4; “The weights 𝑊𝑙∈ ℝ𝐾×𝐾×𝐶𝑖×𝐶𝑜 are the trainable parameters of the layer l, where 𝐾 and 𝐶𝑜 are the kernel dimensions and the number of output channels respectively,” wherein “the number of output channels” is equivalent to a number of filters in a given layer, “l.” III. Autoencoder-Based Low-Rank Filter-Sharing, Autoencoder Training, pp. 4, col. 1, paragraph 3; “If a lot of values in 𝑀prune are zero, a large percentage of filters are pruned. To mitigate this problem, the mask regularization function ℒprune=1/𝐶𝑜Σ|𝑚| is multiplied with a scaling factor 𝑣prune= max (0,1−𝑒(𝑚∗(𝜃−prmax))), which decays with increasing zero fraction in the mask and slows down the pruning rate towards the end of the training,” wherein “the mask regularization function ℒprune” is calculated using the number of filters “𝐶𝑜.” Therefore, the regularizing is based at least in part on a number of filters in a given layer of the initial AE neural network.).
Frickenstein is analogous to the claimed invention as both are directed to filter pruning for computational efficiency in neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the filter pruning method of Kim to include a regularization function calculated using the number of filters, as taught by Frickenstein. The motivation to do so is to create an optimization mechanism for the neural network that incentivizes filter pruning for the purpose of performance (III. Autoencoder-Based Low-Rank Filter-Sharing, Autoencoder Training, pp. 4, col. 1, paragraph 3; “Each autoencoder is trained individually by a dedicated SGD optimizer, referred to as an autoencoder optimizer. The optimization objective for such an optimizer lies in minimizing the loss function ℒae= ℒ𝑟𝑒𝑐+𝑣prune ∙ ℒprune,” wherein the overall loss is greater when the number of filters “𝐶𝑜” is larger. Therefore, the optimization loss decreases the more filters that are pruned.).
Regarding claim 7, the combination of Kim, Park and Frickenstein teaches the computer system of claim 6 (and thus the rejection of claim 6 is incorporated).
Kim does not explicitly teach wherein a subset of the weights associated with the
subset of filters is linearly driven below the predefined value based at least in part on the number of filters in the given layer. However, Frickenstein, in the area of autoencoder-based pruning methods, teaches this limitation (Frickenstein, III. Autoencoder-Based Low-Rank Filter-
Sharing, pp. 3, col. 2, paragraph 3; “In order to dynamically select the most salient filters, an additional trainable parameter, denoted mask 𝑀∈ℝ1×1×1×𝐶𝑜, is introduced with its individual elements 𝑚𝑖∈ 𝑀. By exploiting the sparsity-inducing property of L1 regularization, individual values in the mask M are driven towards zero during training. Since the optimizer usually reaches values close to zero, but not exactly zero, clipping is performed to zero out values that are below a certain threshold t,” thereby indicating that the regularizing is done by driv[ing] a subset of the weights below a predefined value. III. Autoencoder-Based Low-Rank Filter-Sharing, Autoencoder Training, pp. 4, col. 1, paragraph 3; “If a lot of values in 𝑀prune are zero, a large percentage of filters are pruned. To mitigate this problem, the mask regularization function ℒprune=1/𝐶𝑜Σ|𝑚| is multiplied with a scaling factor 𝑣prune=max (0,1−𝑒(𝑚∗(𝜃−prmax))), which decays with increasing zero fraction in the mask and slows down the pruning rate towards the end of the training,” wherein “the mask regularization function ℒ𝑝𝑟𝑢𝑛𝑒” quantifies the loss associated with the number of remaining filters and thereby incentivizes the regularizing or pruning of filters, which is carried out in the manner described above.).
Frickenstein is analogous to the claimed invention as both are directed to filter pruning for computational efficiency in neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the filter pruning method of Kim to reduce and zero out the weights dependent on the number of filters in a given layer, as taught by Frickenstein. The motivation to do so is to have a parameter with which one can customize and control the performance-accuracy balance of a given compressed neural network to meet the requirements of a given system (Frickenstein, IV. Experimental Results, A. Configuration Space Exploration, Setup 3, pp. 5, col. 1, paragraph 1; “Five variants of ALF, resulting in different sparsity rates, are explored in Fig 2c and compared
to the uncompressed Plain-20 (90.5% accuracy). The first three variants differ in terms of the threshold 𝑡∈ {5∙10−5,1∙10−4,5∙10−4} while the learning rate 𝑙𝑟𝑎𝑒=1∙10−3. We observe that the pruning gets more aggressive when the threshold t is increased. The number of non-zero filters remaining are 40.17%, 38.6% and 35.71% respectively (see green, pink and blue curves). We select the threshold 𝑡=1𝑒−4 as a trade-off choice between sparsity and accuracy.” Note that claims 5 and claim 7 both utilize a predefined value to control pruning of the filters. The principle difference is that claim 7 specifies the number of filters in a layer as a factor in the regularization while claim 5 does not. Nonetheless, the motivation to use such a parameter remains the same in both instances: to control the level of pruning consistent with a desired performance-accuracy balance, which is further evidenced by Frickenstein’s use of a loss function.).
Claim 16 is a non-transitory computer-readable storage medium claim corresponding to the steps of claim 5 and is thus rejected for the same reasons as claim 5.
Claim 19 is a method claim corresponding to the steps of claim 5 and is thus rejected for the same reasons as claim 5.
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Park in further view of Liu et al. (“Content-Aware GAN Compression,” hereinafter Liu).
Regarding claim 11, the combination of Kim and Park teaches the computer system of claim 1 (and thus the rejection of claim 1 is incorporated).
Kim does not explicitly teach wherein a difference of an image quality of an output of
the initial AE neural network and the second AE neural network is less than a predefined value. However, Liu, in the area of model compression for image classification tasks, teaches this limitation (Liu, 4.2. Pruning Effectiveness, pp. 12160, cols. 1-2, paragraphs 6-1; “Three baselines are included to show our effectiveness: (1) training from scratch by keeping the pruned network structure while re-initializing the weights; (2) a conventional classification pruning method, to remove channels with low activations [24]; (3) random pruning.” Liu, Table 1;
This table depicts various metrics pertaining to the models evaluated in Liu wherein “FID,” or Fréchet Inception Distance, is a measurement of image quality. The measurements for the “baselines” denote predefined values that are being compared to the result of the “ℓ1-out Pruning” model, or the second…neural network. The difference between the FID measurement of the “Original Full-Size model,” or the initial…neural network, and the “ℓ1-out Pruning” model is less than the difference between the FID measurement of the “Original Full-Size” model and any one of the three “baselines.”).
Liu is analogous to the claimed invention as both are from the same field of endeavor that is, model pruning and transfer learning. Kim assesses the image quality of an output from [an] initial AE neural network but does not explicitly evaluate the difference in this image quality measurement and the image quality of an output from a second AE neural network as being
less than a predefined value. Liu teaches such a method using generative adversarial networks (GANs) (Liu, Abstract; “We first introduce effective channel pruning and knowledge distillation schemes specialized for unconditional GANs. We then propose a novel content-aware method to guide the processes of both pruning and distillation. With content-awareness, we can effectively prune channels that are unimportant to the contents of interest, e.g., human faces, and focus our distillation on these regions, which significantly enhances the distillation quality.”). Figure 4 of the specification of the claimed invention and its associated description at paragraph [023] specify using autoencoder-based GANs as a possible architecture to which the filter pruning regularization techniques can be applied, “FIG. 4 is a drawing illustrating an example of a regularization technique for dynamic channel-filter condensing and pruning Generative Adversarial Network (GAN)-based autoencoders (AEs) in accordance with an embodiment of the present disclosure,” thereby indicating that a generative adversarial network such as the“ℓ1-out Pruning” used by Liu would amount to a simple substitution. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to modify the combined filter pruning and knowledge transfer method of Kim and Park to minimize the difference in image quality between the teacher and student model, as taught by Liu. The motivation to do so is to generate a final model with improved computational efficiency that does not come at the cost of severe performance drop-offs (Liu, 4.2. Pruning Effectiveness, pp. 12160, col. 2, paragraph 2; “As shown in Tab. 1, the low-act pruned model has merely the same FID as training from scratch, even worse than random pruning. This indicates that directly applying classification
pruning metric can fail on GAN compression. Moreover, we find that the ℓ1-out pruned generator has an only 0.9 FID loss from the full-size model with 50% less FLOPs, and it achieves the best FID among compared methods.”).
Regarding claim 12, the combination of Kim, Park and Liu teaches the computer system of claim 11 (and thus the rejection of claim 11 is incorporated).
Kim does not explicitly teach wherein the image quality comprises or corresponds to
a Frechet Inception Distance (FID). However, Liu, in the area of model compression for image classification tasks, teaches this limitation (Liu, 4.1. Evaluation Metrics, pp. 12159, col. 2, paragraph 5; “We use the following five quantitative metrics to evaluate the image generation and image projection performance of a GAN: Inception Score (IS) [42], Fréchet Inception Distance (FID) [21], Perceptual Path Length (PPL) [29], and PSNR/LPIPS between real and projected images.” Liu, 4.1. Evaluation Metrics, Fréchet Inception Distance, pp. 12160, col. 1, paragraph 2; “FID quantitates the similarity between the synthetic images from a generator and the real-world images. It is computed by feed-forwarding two sets of images to an inception network followed by a Fréchet Distance [14] measurement between their corresponding activation features.”).
Liu is analogous to the claimed invention as both are from the same field of endeavor that is, model pruning and transfer learning. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to further modify the image quality evaluation to use Fréchet Inception Distance, as taught by Liu. The motivation to do so is that Fréchet Inception Distance is a metric specifically designed for measuring differences in quality between a generator’s output and ground-truth images (Liu, 4.1. Evaluation Metrics, Fréchet Inception Distance, pp. 12160, col. 1, paragraph 2; “FID quantitates the similarity between the synthetic images from a generator and the real-world images.”).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
The Applicant argues that neither Kim nor Park teach the newly added limitation to independent claims 1, 14 and 17(pages 6-9). The Applicant is directed to the rejection the new grounds of rejection as necessitated by of the newly added limitation in the amendment.
Applicant’s arguments regarding the eligibility of the dependent claims rely on the novelty of the independent claims and are therefore unpersuasive. As such, the rejections made under 35 U.S.C. 103 stand.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Guo 20230004809—Eliminates at least one channel among a plurality of channels included in the first layer on the basis of the acquired function value so as to achieve update to a third layer.
Guo 20220351044 -- Selects on the basis of a preset pruning rule, initial pruned channels and initial unpruned channels of an initial deep neural network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CESAR PAULA whose telephone number is (571)272-4128. The examiner can normally be reached Monday - Friday, 6.30am- 4:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Wiley can be reached at (571)272-3923. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights