DETAILED ACTION
This action is in response to the application filed 05/30/2023. Claims 1-26 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 08/04/2023 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
[0001]: “the entire contents of which is incorporated herein” should be “the entire contents of which are incorporated herein”
[0054]: “while measuring the parameters importance” should be “while measuring the parameters’ importance”
[0061]: “to look ahead the performance if growing” is improper grammar.
Appropriate correction is required.
Drawings
The drawings are objected to because figures 3 and 4 have fuzzy, difficult to parse text and diagrams. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-26 are rejected under 35 U.S.C. 101 because the claimed inventions are directed to non-statutory subject matter without significantly more.
Claim 1
Step 1: The claim recites “A method”, and is therefore directed to the statutory category of process
Step 2A Prong 1: The claim recites the following judicial exception(s)
estimating an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This can be performed as a mental process. One can mentally gauge the importance of the temporarily pruned subset of parameters.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the following additional element(s)
at a device, in an iteration of at least one iteration of a neural network model sparsification process: This is mere instruction to execute recited judicial exceptions in a generic computing device (MPEP 2106.05(f)).
training an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned: Pruning neural network parameters is a conventional technique in machine learning and thus amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
estimating an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This is mere instruction to execute a judicial exception based on training and frozen parameters in a generic manner (MPEP 2106.05(f)).
updating the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The following additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
at a device, in an iteration of at least one iteration of a neural network model sparsification process: This is mere instruction to execute recited judicial exceptions in a generic computing device (MPEP 2106.05(f)).
training an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned: Pruning neural network parameters is well-known technique to reduce the computational requirements of a neural network, as noted by Wagle et al. ("CONFIGURABLE BNN ASIC USING A NETWORK OF PROGRAMMABLE THRESHOLD LOGIC STANDARD CELLS", filed 10/18/2021, US 20220121915 A1): “Regardless of the hardware platform (e.g., central processing unit (CPU), graphical processing unit (GPU), field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC)) on which DNNs are deployed, the biggest challenge to improving their performance and energy efficiency has been the on-chip storage requirement. Cost and yield considerations limit the feasible on-chip storage to be one to two orders of magnitude smaller than what is required by many of the popular DNN models, forcing most of the parameters for even moderate size DNN s to be stored in off-chip dynamic random-access memory (DRAM). This results in large energy (>200x) and delay (> 10x) penalties. This has accelerated efforts to drastically reduce the DRAM storage requirements and the associated access delays. Some well-known methods include weight and synapse pruning, quantization (i.e., reducing bit widths of inputs and weight), weight sharing, Huffman coding, and approximate arithmetic, to name a few” (Wagle, [0004])
estimating an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This is mere instruction to execute a judicial exception based on training and frozen parameters in a generic manner (MPEP 2106.05(f)).
updating the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 2
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
where the device further: prunes the subset of parameters from the neural network model prior to training the active set of parameters in the neural network model: Pruning neural network parameters is still a conventional technique in machine learning and thus amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
where the device further: prunes the subset of parameters from the neural network model prior to training the active set of parameters in the neural network model: Pruning neural network parameters is still a well-known technique to reduce the computational requirements of a neural network, as noted by Wagle et al. ("CONFIGURABLE BNN ASIC USING A NETWORK OF PROGRAMMABLE THRESHOLD LOGIC STANDARD CELLS", filed 10/18/2021, US 20220121915 A1): “This has accelerated efforts to drastically reduce the DRAM storage requirements and the associated access delays. Some well-known methods include weight and synapse pruning, quantization (i.e., reducing bit widths of inputs and weight), weight sharing, Huffman coding, and approximate arithmetic, to name a few” (Wagle, [0004])
Claim 3
Step 1: The claim recites a process, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the pruning is unstructured: Unstructured pruning is a conventional technique in neural networks, thus this amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the pruning is unstructured: Unstructured pruning is a popular technique in the field of neural networks, as noted by Varma et al. (Method And System For Dynamic Compositional General Continual Learning, filed 6/29/2022, US 20230385644 A1): “Network pruning [35], a popular method for compressing DNNs, can be seen as an indirect attempt at mimicking modularity and sparsity in the human brain, by extracting a sub-network of the DNN that is primarily responsible for the task at hand. Pruning is generally achieved through removing unimportant connections such as weights with low magnitudes—called unstructured pruning” (Varma, [0011])
Claim 4
Step 1: The claim recites a process, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the pruning is structured: Structured pruning is a conventional technique in neural networks, thus this amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the pruning is structured: Structured pruning is a popular technique in the field of neural networks, as noted by Varma et al. (Method And System For Dynamic Compositional General Continual Learning, filed 6/29/2022, US 20230385644 A1): “Network pruning [35], a popular method for compressing DNNs, can be seen as an indirect attempt at mimicking modularity and sparsity in the human brain, by extracting a sub-network of the DNN that is primarily responsible for the task at hand. Pruning is generally achieved through removing unimportant connections such as weights with low magnitudes—called unstructured pruning, or removing unimportant structures such as unimportant channels, filters, or layers—called structured pruning [35]” (Varma, [0011])
Claim 5
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein a number of active parameters to be used during each iteration of the at least one iteration of the neural network model sparsification process is predefined: Pruning parameters from the active set of parameters is still an instance of pruning, a conventional technique, thus this is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein a number of active parameters to be used during each iteration of the at least one iteration of the neural network model sparsification process is predefined: Pruning parameters from the active set of parameters is still an instance of pruning, a conventional technique, as noted by Wagle et al. ("CONFIGURABLE BNN ASIC USING A NETWORK OF PROGRAMMABLE THRESHOLD LOGIC STANDARD CELLS", filed 10/18/2021, US 20220121915 A1): “This has accelerated efforts to drastically reduce the DRAM storage requirements and the associated access delays. Some well-known methods include weight and synapse pruning, quantization (i.e., reducing bit widths of inputs and weight), weight sharing, Huffman coding, and approximate arithmetic, to name a few” (Wagle, [0004])
Claim 6
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the neural network model is a sparse neural network model having a predefined number of active parameters: Making a network model sparse through pruning is still a conventional technique, and thus this amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the neural network model is a sparse neural network model having a predefined number of active parameters: Making a network model sparse through pruning is a popular technique in the field, as noted by Varma et al. (Method And System For Dynamic Compositional General Continual Learning, filed 6/29/2022, US 20230385644 A1): “Network pruning [35], a popular method for compressing DNNs, can be seen as an indirect attempt at mimicking modularity and sparsity in the human brain, by extracting a sub-network of the DNN that is primarily responsible for the task at hand. Pruning is generally achieved through removing unimportant connections such as weights with low magnitudes—called unstructured pruning, or removing unimportant structures such as unimportant channels, filters, or layers—called structured pruning [35]” (Varma, [0011])
Claim 7
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the active set of parameters are randomly selected for an initial iteration of the neural network model sparsification process: Randomly pruning parameters from a neural network is a conventional technique, thus this is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the active set of parameters are randomly selected for an initial iteration of the neural network model sparsification process: Randomly pruning parameters from a neural network is a conventional technique, as noted by Wu et al. (PRUNING AND ACCELERATING NEURAL NETWORKS WITH HIERARCHICAL FINE-GRAINED STRUCTURED SPARSITY, filed 2/28/2022, US 20230062503 A1): “A conventional pruning method, unstructured pruning randomly removes individual non-zero values from the weight tensor, resulting in an unpredictable and unstructured distribution of zeros” (Wu, [0003])
Claim 8
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein for each iteration of the at least one iteration of the neural network model sparsification process, a mask defines the active set of parameters and the subset of parameters pruned from the neural network model: Masking model elements using a sparsity mask is a conventional technique, thus this is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein for each iteration of the at least one iteration of the neural network model sparsification process, a mask defines the active set of parameters and the subset of parameters pruned from the neural network model: Masking model elements using a sparsity mask is a conventional technique, as noted by Liu et al. (IMAGE PROCESSING METHOD, AN IMAGE PROCESSING APPARATUS, AND A SURVEILLANCE SYSTEM, published 12/17/2020, US 20200394418 A1): “It should be understood that in the conventional sparse convolution neural network, it determines whether a region is sparse based on a block, and convolution calculation is performed based on a sparse block, however, in the above described processing process, the block mask still contains a large amount of sparseness, which results in a large amount of computation.” (Liu, [0047])
Claim 9
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein estimating the importance of the subset of parameters pruned from the neural network model further includes re-activating the subset of parameters in the neural network model, and wherein the freezing and the re-activating is performed by updating the mask: This is mere instruction to execute a judicial exception based on freezing and reactivation of network parameters in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein estimating the importance of the subset of parameters pruned from the neural network model further includes re-activating the subset of parameters in the neural network model, and wherein the freezing and the re-activating is performed by updating the mask: This is mere instruction to execute a judicial exception based on freezing and reactivation of network parameters in a generic manner (MPEP 2106.05(f)).
Claim 10
Step 1: The claim recites a process, as in claim 9
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the subset of parameters are re-activated with their most recently used value: Estimating the importance of the subset of parameters based on training the subset is still mere instruction to execute a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the subset of parameters are re-activated with their most recently used value: Estimating the importance of the subset of parameters based on training the subset is still mere instruction to execute a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 11
Step 1: The claim recites a process, as in claim 8
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein updating the active set of parameters in the neural network model is performed by defining the updated active set of parameters in the mask: Updating the active set based on the importance is still mere instruction to update based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein updating the active set of parameters in the neural network model is performed by defining the updated active set of parameters in the mask: Updating the active set based on the importance is still mere instruction to update based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 12
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the active set of parameters are trained over a first plurality of iterations to stabilize the neural network model and to exploit the neural network model to improve its performance with respect to a defined performance goal, and wherein the subset of parameters are trained over a second plurality of iterations with an assumption of stability of the neural network model and to exploit the neural network model to maximize its performance with respect to the defined performance goal: This is mere instruction to train the active parameters to improve performance in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the active set of parameters are trained over a first plurality of iterations to stabilize the neural network model and to exploit the neural network model to improve its performance with respect to a defined performance goal, and wherein the subset of parameters are trained over a second plurality of iterations with an assumption of stability of the neural network model and to exploit the neural network model to maximize its performance with respect to the defined performance goal: This is mere instruction to train the active parameters to improve performance in a generic manner (MPEP 2106.05(f)).
Claim 13
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites the following further judicial exception(s)
wherein an importance of the parameters in the active set of parameters is additionally estimated: This can be performed as a mental process. One can mentally gauge the importance of the active set of parameters.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the additional element(s)
Step 2B: The additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
Claim 14
Step 1: The claim recites a process, as in claim 13
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the active set of parameters is further updated, based on the importance of the parameters in the active set of parameters: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the active set of parameters is further updated, based on the importance of the parameters in the active set of parameters: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 15
Step 1: The claim recites a process, as in claim 14
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the active set of parameters are updated to include a defined number of parameters with highest importance from among the active set of parameters and the subset of parameters: Dropping low-scoring parameters in a network pruning process is a conventional technique, thus this is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the active set of parameters are updated to include a defined number of parameters with highest importance from among the active set of parameters and the subset of parameters: Dropping low-scoring parameters in a network pruning process is a conventional technique, as noted by Tanaka et al. (System And Method For Pruning Neural Networks At Initialization Using Iteratively Conserving Synaptic Flow, filed 7/15/2021, US 12406185 B1): “Neural network pruning may be done after training or before training. When the pruning is performed after training, conventional pruning algorithms assign scores to parameters in neural networks after training and remove the parameters with the lowest scores [5], [22], [23]” (Tanaka, column 1, paragraph 7)
Claim 16
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein updating the active set of parameters includes growing the active set of parameters with one or more of the parameters in the subset of parameters previously pruned from the neural network model: Updating the active set of parameters based on the importance of the subset of parameters is still mere instruction to update the active set based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein updating the active set of parameters includes growing the active set of parameters with one or more of the parameters in the subset of parameters previously pruned from the neural network model: Updating the active set of parameters based on the importance of the subset of parameters is still mere instruction to update the active set based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 17
Step 1: The claim recites a process, as in claim 1
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
performs an additional iteration of the neural network model sparsification process, based on the updated active set of parameters: Iterative pruning is a well-known technique in the field, thus this is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
performs an additional iteration of the neural network model sparsification process, based on the updated active set of parameters: Iterative pruning is a well-known technique, as noted by Jeong et al. (EFFECTIVE NETWORK COMPRESSION USING SIMULATION-GUIDED ITERATIVE PRUNING, published 12/23/2021, US 20210397962 A1): “among network compression methods, an iterative pruning is one of the most popular methods that have proven to be effective in several previous studies, including state of the art methods. In an iterative pruning process, first, importance of weighted values is estimated in an original network, and then, the weighted values having low importance are removed by retraining the rest weighted values through fine adjustment. Such pruning process is iteratively performed until a stop condition is met.” (Jeong, [0003])
Claim 18
Step 1: The claim recites “A system”, and is therefore directed to the statutory category of machine
Step 2A Prong 1: The claim recites the following judicial exception(s)
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This can be performed as a mental process. One can mentally gauge the importance of the temporarily pruned subset of parameters.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the following additional element(s)
a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions, in an iteration of at least one iteration of a neural network model sparsification process: This is mere instruction to execute recited judicial exceptions in a generic computing device (MPEP 2106.05(f)).
train an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned: Pruning neural network parameters is a conventional technique in machine learning and thus amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This is mere instruction to execute a judicial exception based on training and frozen parameters in a generic manner (MPEP 2106.05(f)).
update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The following additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions, in an iteration of at least one iteration of a neural network model sparsification process: This is mere instruction to execute recited judicial exceptions in a generic computing device (MPEP 2106.05(f)).
train an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned: Pruning neural network parameters is well-known technique to reduce the computational requirements of a neural network, as noted by Wagle et al. ("CONFIGURABLE BNN ASIC USING A NETWORK OF PROGRAMMABLE THRESHOLD LOGIC STANDARD CELLS", filed 10/18/2021, US 20220121915 A1): “Regardless of the hardware platform (e.g., central processing unit (CPU), graphical processing unit (GPU), field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC)) on which DNNs are deployed, the biggest challenge to improving their performance and energy efficiency has been the on-chip storage requirement. Cost and yield considerations limit the feasible on-chip storage to be one to two orders of magnitude smaller than what is required by many of the popular DNN models, forcing most of the parameters for even moderate size DNN s to be stored in off-chip dynamic random-access memory (DRAM). This results in large energy (>200x) and delay (> 10x) penalties. This has accelerated efforts to drastically reduce the DRAM storage requirements and the associated access delays. Some well-known methods include weight and synapse pruning, quantization (i.e., reducing bit widths of inputs and weight), weight sharing, Huffman coding, and approximate arithmetic, to name a few” (Wagle, [0004])
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This is mere instruction to execute a judicial exception based on training and frozen parameters in a generic manner (MPEP 2106.05(f)).
update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claims 19-25
Step 1: Claims 19-25 recite a machine, as in claim 18.
Step 2A Prong 1: Claims 19-25 recite the same judicial exception(s) as claims 2, 8-9, 12, and 16-17, respectively.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through any additional elements. The analysis of claims 19-25 at this step mirrors that of claims 2, 8-9, 12, and 16-17, respectively, with the exception that claims 19-25 are directed to “a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions”, said instructions containing operations mirroring those of claims 2, 8-9, 12, and 16-17. This is a mere instruction to apply the exceptions using generic computer equipment (MPEP 2106.05(f)).
Step 2B: The additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s). The analysis of claims 19-25 at this step mirrors that of claims 2, 8-9, 12, and 16-17, with the exception that claims 19-25 are directed to “a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions”, said instructions containing operations mirroring those of claims 2, 8-9, 12, and 16-17. This is mere instruction to apply the exceptions using generic computer equipment (MPEP 2106.05(f)).
Claim 26
Step 1: The claim recites “A non-transitory computer-readable media”, and is therefore directed to the statutory category of article of manufacture
Step 2A Prong 1: The claim recites the following judicial exception(s)
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This can be performed as a mental process. One can mentally gauge the importance of the temporarily pruned subset of parameters.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the following additional element(s)
A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device, in an iteration of at least one iteration of a neural network model sparsification process: This is mere instruction to execute recited judicial exceptions in a generic computing device (MPEP 2106.05(f)).
train an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned: Pruning neural network parameters is a conventional technique in machine learning and thus amounts to insignificant extra-solution activity (MPEP 2106.05(g)).
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This is mere instruction to execute a judicial exception based on training and frozen parameters in a generic manner (MPEP 2106.05(f)).
update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The following additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device, in an iteration of at least one iteration of a neural network model sparsification process: This is mere instruction to execute recited judicial exceptions in a generic computing device (MPEP 2106.05(f)).
train an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned: Pruning neural network parameters is well-known technique to reduce the computational requirements of a neural network, as noted by Wagle et al. ("CONFIGURABLE BNN ASIC USING A NETWORK OF PROGRAMMABLE THRESHOLD LOGIC STANDARD CELLS", filed 10/18/2021, US 20220121915 A1): “Regardless of the hardware platform (e.g., central processing unit (CPU), graphical processing unit (GPU), field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC)) on which DNNs are deployed, the biggest challenge to improving their performance and energy efficiency has been the on-chip storage requirement. Cost and yield considerations limit the feasible on-chip storage to be one to two orders of magnitude smaller than what is required by many of the popular DNN models, forcing most of the parameters for even moderate size DNN s to be stored in off-chip dynamic random-access memory (DRAM). This results in large energy (>200x) and delay (> 10x) penalties. This has accelerated efforts to drastically reduce the DRAM storage requirements and the associated access delays. Some well-known methods include weight and synapse pruning, quantization (i.e., reducing bit widths of inputs and weight), weight sharing, Huffman coding, and approximate arithmetic, to name a few” (Wagle, [0004])
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: This is mere instruction to execute a judicial exception based on training and frozen parameters in a generic manner (MPEP 2106.05(f)).
update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: This is mere instruction to update the active set of parameters based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-26 are rejected under 35 U.S.C. 103 as being unpatentable over Ma et al. (EFFECTIVE MODEL SPARSIFICATION BY SCHEDULED GROW-AND-PRUNE METHODS, published 3/4/2022, arXiv:2106.09857v3), hereafter referred to as Ma, in view of Yao et al. (DYNAMIC NEURAL NETWORK SURGERY, published 2019, US 2019/0188567 A1), hereafter referred to as Yao, and further in view of Tan et al. (FREEZE-OUT AS A REGULARIZER IN TRAINING NEURAL NETWORKS, published 2021, US 2021/0232909 A1), hereafter referred to as Tan.
Regarding claim 1, Ma discloses [a] method, comprising: at a device, in an iteration of at least one iteration of a neural network model sparsification process:
training an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned:
“During model training, one of the sparse partitions is grown to dense while the rest remain sparse. After some epochs of training, the previously dense partition is pruned to sparse, and an alternate partition is grown to dense. This process is repeated (iterated) so that all partitions are grown to dense and pruned back to sparse multiple times” (Ma, page 2, paragraph 3)
PNG
media_image1.png
401
1162
media_image1.png
Greyscale
(Ma, page 2, Figure 1)
“As shown in Figure 1, the cyclic GaP (C-GaP) method rotates the dense partitions among all K partitions (K = 4 in Figure 1). Starting from all partitions with random sparse masks, the first partition is grown to a dense one. All weights in the first partition and the masked weights in the remaining 3 partitions are trained for a few epochs (Step 1). Then, the first partition is pruned to sparse (Step 2). Next, we apply the same strategy to the second partition, then continue iterating over all K partitions. When all layers are cyclically grown and pruned once after K steps, we call it one round. This process is repeated K times.” (Ma, page 3, paragraph 2). Each partition is grown and pruned K times (with prunings prior to the final round being temporary).
“In the scheduled GaP methods, all weights (parameters) to be grown (or to be pruned) belong to the same partition. All weights in the model are explored when every partition is grown to dense and pruned to sparse.” (Ma, page 2, paragraph 4)
estimating an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model; and updating the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: “All of our experimental results are trained and inferenced using PyTorch in the machines with 8 NVIDIA-V100 GPUs. The cyclic GaP is trained in one training node and the parallel GaP is trained in K training nodes, where K is the number of model partitions. In the pruning stage of the GaP method, the weights with the lowest magnitudes (importance) are pruned to zero.” (Ma, page 5, paragraph 2). The active set is updated to include only connections with greater importance.
Ma relates to dynamic pruning methods for neural networks and is analogous to the claimed invention.
While Ma fails to disclose the further limitations of the claim Yao discloses a method of estimating an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model; and updating the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: “The compression discussed herein may include iterative pruning and splicing operations and parameter weight update operations. Such pruning operations (e.g., disconnecting an available connection at a particular iteration) may compress the DNN model by removing unimportant connections and such splicing operations (e.g., reconnecting previously disconnected available connections at a particular iteration) may provide recovery for pruned connections that are found to be important over the iterations (training)” (Yao, [0021]). Pruned parameters found to be important are added back to the active set.
Yao relates to dynamic pruning and splicing methods for neural networks and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ma to return high-importance pruned parameters to the active set of parameters, as disclosed by Ma. This combination of pruning and splicing can provide sparse neural networks with large compression rates without losing significant accuracy. See Yao, [0022].
While Yao fails to disclose the further limitations of the claim, Tan discloses a method of estimating an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: “In certain embodiments, the freeze-out component 108 can freeze the selected subset of units of the neural network so that weights of output connections from the frozen subset of units will not be updated for a training run. For example, when the freeze-out component 108 can freeze the selected subset of units of the neural network, the connections of frozen units to units of the following layer or layers are not removed or changed. The output of frozen units is not included for a training run but the weights of output connections from the frozen units to units of the following layer or layers are not changed for the training run. Thus, there is no need to update weights of output connections from frozen units to units of following layer(s) during each training run. When units are frozen by the freeze-out component 108 as opposed to being removed when utilizing dropout, the architecture of the neural network remains unchanged. This eliminates a step of updating the weights of output connections from dropped units when utilizing dropout.” (Tan, [0034]). One subset of parameters is frozen for training, while the other is trained and updated as normal.
Tan relates to selective freezing of neural network parameters and is analogous to the claimed invention. The existing combination teaches a method of identifying and unpruning important parameters initially pruned but later found to be important during training. The claimed invention improves upon this method by freezing the active set of parameters while the pruned subset is trained. Tan teaches a method of freezing a subset of a neural network’s parameters, applicable to the existing combination. A person of ordinary skill in the art would have recognized that freezing the active parameters while evaluating pruned parameters would lead to the predictable result of only training the pruned parameters during this phase, and would improve the known device by increasing the efficiency of training at this phase by only training the parameters being evaluated for importance (the pruned parameters) (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Regarding claim 2, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, where the device further: prunes the subset of parameters from the neural network model prior to training the active set of parameters in the neural network model:
PNG
media_image1.png
401
1162
media_image1.png
Greyscale
”Overview of the cyclic GaP (C-GaP) training method. We assume 4 partitions in a model are grown and pruned in a cyclic order. Only one partition is kept dense during training. After K steps, the dense partition is pruned and the whole model is fine-tuned to obtain the sparse model. “ (Ma, page 2, Figure 1).
PNG
media_image2.png
256
712
media_image2.png
Greyscale
(Ma, Figure 2)
Examiner’s note: A partition is pruned in one round of training before growing & training happen in a following round. In figure 1, the first partition is pruned in step 2, followed by regrowth of dense connections and training in step 5.
Regarding claim 3, the rejection of claim 2 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the pruning is unstructured: “The GaP methods are carefully scheduled to explore all weights efficiently, which is not guaranteed in the existing mask exploration methods. We illustrate their differences by an example in Figure 2. In the scheduled GaP methods, all weights to be grown (or to be pruned) belong to the same partition. All weights in the model are explored when every partition is grown to dense and pruned to sparse.” (Ma, page 2, paragraph 4). As stated in paragraph [0025], unstructured pruning is defined by pruning individual weights.
Regarding claim 4, the rejection of claim 2 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the pruning is structured: “MetaPruning (Liu et al., 2019a) use complicated rules to generate the sparsity distribution in the channel level” (Ma, page 8, paragraph 3). As stated in paragraph [0025] of the instant specification, structured pruning an entail pruning channels.
Regarding claim 5, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein a number of active parameters to be used during each iteration of the at least one iteration of the neural network model sparsification process is predefined:
“Then the grow and prune process is iterated K times (Line 4). First, the indices of the partitions to grow and prune are calculated (Line 5). They are determined in a cyclic order. Then, the selected dense partition is pruned to sparse (Line 6). It prunes the partition
θ
S
j
with a pruning ratio r” (Ma, page 4, paragraph 2)
PNG
media_image3.png
141
905
media_image3.png
Greyscale
(Ma, page 6, Table 2)
Examiner’s note: A predefined pruning ratio determines the proportion of parameters to prune. Multiplied by the number of connections, it defines the number of active parameters to be used (not pruned)
Regarding claim 6, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the neural network model is a sparse neural network model having a predefined number of active parameters:
“Then the grow and prune process is iterated K times (Line 4). First, the indices of the partitions to grow and prune are calculated (Line 5). They are determined in a cyclic order. Then, the selected dense partition is pruned to sparse (Line 6). It prunes the partition
θ
S
j
with a pruning ratio r” (Ma, page 4, paragraph 2)
PNG
media_image3.png
141
905
media_image3.png
Greyscale
(Ma, page 6, Table 2)
Examiner’s note: A predefined pruning ratio determines the proportion of parameters to prune. Multiplied by the number of connections, it defines the number of active parameters to be used (not pruned)
Regarding claim 7, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the active set of parameters are randomly selected for an initial iteration of the neural network model sparsification process: “In this section, we describe the scheduled grow-and-prune (GaP) methods in detail. The process starts from a randomly initialized model. First, its weights are pruned randomly to reach the target sparsity, i.e. a random sparse mask is applied to the weights.” (Ma, page 3, paragraph 1).
Regarding claim 8, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein for each iteration of the at least one iteration of the neural network model sparsification process, a mask defines the active set of parameters and the subset of parameters pruned from the neural network model:
“m is a binary mask
m
∈
{
0,1
}
M
, where a zero in m means that the corresponding weight in
θ
is fixed to be zero and a one in m means that the corresponding weight in
θ
is free to be updated.” (Ma, page 3, paragraph 3)
PNG
media_image4.png
401
936
media_image4.png
Greyscale
(Ma, page 3, Algorithm 1). As can be seen in lines 6-7, the sparsity masks are updated and used in each iteration.
Regarding claim 9, the rejection of claim 8 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method,
wherein estimating the importance of the subset of parameters pruned from the neural network model further includes re-activating the subset of parameters in the neural network model:
“The scheduled GaP methodology divides a DNN into several partitions composed of contiguous layers. During model training, one of the sparse partitions is grown to dense while the rest remain sparse. After some epochs of training, the previously dense partition is pruned to sparse, and an alternate partition is grown to dense. This process is repeated so that all partitions are grown to dense and pruned back to sparse multiple times.” (Ma, page 2, paragraph 3). Training of a dense layer, which entails activations of all synapses in the layer, precedes synapse pruning.
PNG
media_image5.png
316
870
media_image5.png
Greyscale
(Ma, page 3, Figure 2)
“In the pruning stage of the GaP method, the weights with the lowest magnitudes (importance) are pruned to zero.” (Ma, page 5, paragraph 2)
and wherein the freezing and the re-activating is performed by updating the mask:
“Algorithm 1 describes this process in more detail. In Line 1, we use f(x;
θ
) to represent an L-layer deep neural network model, where
θ
ϵ
R
M
are M trainable weights and x are training samples. A sparse model is denoted as f(x;_ _ m), where _ denotes element-wise multiplication. m is a binary mask m 2 f0; 1gM, where a zero in m means that the corresponding weight in _ is fixed to be zero and a one in m means that the corresponding weight in _ is free to be updated.” (Ma, page 3, paragraph 3)
“m is a binary mask
m
∈
{
0,1
}
M
, where a zero in m means that the corresponding weight in
θ
is fixed to be zero and a one in m means that the corresponding weight in
θ
is free to be updated.” (Ma, page 3, paragraph 3)
Regarding claim 10, the rejection of claim 9 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the subset of parameters are re-activated with their most recently used value:
“As shown in Figure 1, the cyclic GaP (C-GaP) method rotates the dense partitions among all K partitions (K = 4 in Figure 1). Starting from all partitions with random sparse masks, the first partition is grown to a dense one. All weights in the first partition and the masked weights in the remaining 3 partitions are trained for a few epochs (Step 1). Then, the first partition is pruned to sparse (Step 2). Next, we apply the same strategy to the second partition, then continue iterating over all K partitions. When all layers are cyclically grown and pruned once after K steps, we call it one round. This process is repeated K times” (Ma, page 3, paragraph 2). Each partition is grown and pruned K times.
“Algorithm 1 describes this process in more detail. In Line 1, we use f(x;
θ
) to represent an L-layer deep neural network model, where
θ
ϵ
R
M
are M trainable weights and x are training samples. A sparse model is denoted as f(x;
θ
⊙
m
), where
⊙
denotes element-wise multiplication. m is a binary mask
m
∈
{
0,1
}
M
, where a zero in m means that the corresponding weight in
θ
is fixed to be zero and a one in m means that the corresponding weight in
θ
is free to be updated.” (Ma, page 3, paragraph 3). Pruned weights are masked by multiplying them by 0 during training, thus their weight values don’t change. Unpruned weights are multiplied by 1, thus their weight values can change during training.
Examiner’s note: When previously pruned weights are grown to dense during another round, their mask values change from 0 to 1. Since their values weren’t altered while the mask values were 0, they’re restored to their most recently used values for training.
Regarding claim 11, the rejection of claim 8 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein updating the active set of parameters in the neural network model is performed by defining the updated active set of parameters in the mask:
“the sparse mask of a layer is updated after exploring all weights in the same layer, resulting in better mask-update efficiency” (Ma, page 2, paragraph 2)
PNG
media_image6.png
144
778
media_image6.png
Greyscale
(Ma, page 3, Algorithm 1)
Regarding claim 12, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the active set of parameters are trained over a first plurality of iterations to stabilize the neural network model and to exploit the neural network model to improve its performance with respect to a defined performance goal, and wherein the subset of parameters are trained over a second plurality of iterations with an assumption of stability of the neural network model and to exploit the neural network model to maximize its performance with respect to the defined performance goal:
“In our proofs, we find that cycling through all partitions so that all weights can be trained per round is critical to guarantee the convergence (stability) of C-GaP. Note that previous works update weights either greedily (e.g., RigL (Evci et al., 2020)) or randomly (e.g., SET (Mocanu et al., 2018) and DSR (Mostafa & Wang, 2019)). It may take numerous training steps for these algorithms to have each weight explored and updated. This may explain why C-GaP achieves better accuracy (performance goal) than RigL, SET, and DSR (see Table 1). This intuition is consistent with (Gurbuzbalaban et al., 2019; Ying et al., 2018) which theoretically establish that sampling data cyclically converges faster than uniformly randomly in SGD training.” (Ma, page 7, paragraph 5).
“If the learning rate
γ
=
1
/
(
4
K
L
T
Q
)
, the sparse models generated by the C-GaP method after Q rounds will converge as follows (assumption of stability):
PNG
media_image7.png
110
791
media_image7.png
Greyscale
” (Ma, page 7, Equation 1)
Regarding claim 13, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein an importance of the parameters in the active set of parameters is additionally estimated: “All of our experimental results are trained and inferenced using PyTorch in the machines with 8 NVIDIA-V100 GPUs. The cyclic GaP is trained in one training node and the parallel GaP is trained in K training nodes, where K is the number of model partitions. In the pruning stage of the GaP method, the weights with the lowest magnitudes (importance) are pruned to zero … For uniform sparsity, we prune individual layer separately by sorting all weights in the layer based on their magnitudes, and prune away any weight whose magnitude is below the percentile described as the sparsity level. Thus, all layers receive the same sparsity. For non-uniform sparsity, the pruning process is similar, but the weights in the entire model are sorted together” (Ma, page 5, paragraph 2).” (Ma, page 5, paragraph 2). All weights in a layer are sorted to determine relative importance, necessitating the measurement of importance on weights that aren’t ultimately pruned (active set).
Regarding claim 14, the rejection of claim 13 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the active set of parameters is further updated, based on the importance of the parameters in the active set of parameters: “All of our experimental results are trained and inferenced using PyTorch in the machines with 8 NVIDIA-V100 GPUs. The cyclic GaP is trained in one training node and the parallel GaP is trained in K training nodes, where K is the number of model partitions. In the pruning stage of the GaP method, the weights with the lowest magnitudes (importance) are pruned to zero.” (Ma, page 5, paragraph 2). The active set is updated to include only connections with greater importance.
Regarding claim 15, the rejection of claim 14 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the active set of parameters are updated to include a defined number of parameters with highest importance from among the active set of parameters and the subset of parameters:
“Then the grow and prune process is iterated K times (Line 4). First, the indices of the partitions to grow and prune are calculated (Line 5). They are determined in a cyclic order. Then, the selected dense partition is pruned to sparse (Line 6). It prunes the partition
θ
S
j
with a pruning ratio r” (Ma, page 4, paragraph 2)
PNG
media_image3.png
141
905
media_image3.png
Greyscale
(Ma, page 6, Table 2)
Examiner’s note: A predefined pruning ratio determines the proportion of parameters to prune. Multiplied by the number of connections, it defines the number of parameters with highest importance to include in the active set. As noted in the rejection for parent claim 14, the most important weights are included in the active set.
Regarding claim 16, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein updating the active set of parameters includes growing the active set of parameters with one or more of the parameters in the subset of parameters previously pruned from the neural network model:
“This process is repeated so that all partitions are grown to dense and pruned back to sparse multiple times.” (Ma, page 2, paragraph 3)
PNG
media_image8.png
162
865
media_image8.png
Greyscale
(Ma, page 3, Figure 2). In this example, some previously pruned connections are later made parts of the active set.
Regarding claim 17, the rejection of claim 1 in view of Ma, Yao, and Tan is incorporated. Ma further discloses a method, wherein the device further: performs an additional iteration of the neural network model sparsification process, based on the updated active set of parameters:
PNG
media_image1.png
401
1162
media_image1.png
Greyscale
”Overview of the cyclic GaP (C-GaP) training method. We assume 4 partitions in a model are grown and pruned in a cyclic order. Only one partition is kept dense during training. After K steps, the dense partition is pruned and the whole model is fine-tuned to obtain the sparse model.” (Ma, page 2, Figure 1)
Regarding claim 18, Ma discloses instructions, in an iteration of at least one iteration of a neural network model sparsification process, to:
train an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned:
“During model training, one of the sparse partitions is grown to dense while the rest remain sparse. After some epochs of training, the previously dense partition is pruned to sparse, and an alternate partition is grown to dense. This process is repeated (iterated) so that all partitions are grown to dense and pruned back to sparse multiple times” (Ma, page 2, paragraph 3)
PNG
media_image1.png
401
1162
media_image1.png
Greyscale
(Ma, page 2, Figure 1)
“As shown in Figure 1, the cyclic GaP (C-GaP) method rotates the dense partitions among all K partitions (K = 4 in Figure 1). Starting from all partitions with random sparse masks, the first partition is grown to a dense one. All weights in the first partition and the masked weights in the remaining 3 partitions are trained for a few epochs (Step 1). Then, the first partition is pruned to sparse (Step 2). Next, we apply the same strategy to the second partition, then continue iterating over all K partitions. When all layers are cyclically grown and pruned once after K steps, we call it one round. This process is repeated K times.” (Ma, page 3, paragraph 2). Each partition is grown and pruned K times (with prunings prior to the final round being temporary).
“In the scheduled GaP methods, all weights (parameters) to be grown (or to be pruned) belong to the same partition. All weights in the model are explored when every partition is grown to dense and pruned to sparse.” (Ma, page 2, paragraph 4)
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model; and update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: “All of our experimental results are trained and inferenced using PyTorch in the machines with 8 NVIDIA-V100 GPUs. The cyclic GaP is trained in one training node and the parallel GaP is trained in K training nodes, where K is the number of model partitions. In the pruning stage of the GaP method, the weights with the lowest magnitudes (importance) are pruned to zero.” (Ma, page 5, paragraph 2). The active set is updated to include only connections with greater importance.
Ma relates to dynamic pruning methods for neural networks and is analogous to the claimed invention.
While Ma fails to disclose the further limitations of the claim Yao discloses [a] system, comprising: a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions: “The material disclosed herein may also be implemented as instructions stored on a machine-readable medium (memory storage), which may be read and executed by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM) (non-transitory memory storage)” (Yao, [0018]
Yao further discloses instructions to estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model; and update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: “The compression discussed herein may include iterative pruning and splicing operations and parameter weight update operations. Such pruning operations (e.g., disconnecting an available connection at a particular iteration) may compress the DNN model by removing unimportant connections and such splicing operations (e.g., reconnecting previously disconnected available connections at a particular iteration) may provide recovery for pruned connections that are found to be important over the iterations (training)” (Yao, [0021]). Pruned parameters found to be important are added back to the active set.
Yao relates to dynamic pruning and splicing methods for neural networks and is analogous to the claimed invention.
Ma teaches a method of neural network sparsification. The claimed invention improves upon this method by storing it in the form of instructions on computer hardware. Yao teaches computer hardware for a neural network sparsification system, applicable to Ma. A person of ordinary skill in the art would have recognized that storing Ma’s method as computer instructions on Yao’s hardware would lead to the predictable result of the method being executable by a computing system, and would improve the known device by allowing it to be performed with real data (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ma to return high-importance pruned parameters to the active set of parameters, as disclosed by Ma. This combination of pruning and splicing can provide sparse neural networks with large compression rates without losing significant accuracy. See Yao, [0022].
While Yao fails to disclose the further limitations of the claim, Tan discloses instructions to estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: “In certain embodiments, the freeze-out component 108 can freeze the selected subset of units of the neural network so that weights of output connections from the frozen subset of units will not be updated for a training run. For example, when the freeze-out component 108 can freeze the selected subset of units of the neural network, the connections of frozen units to units of the following layer or layers are not removed or changed. The output of frozen units is not included for a training run but the weights of output connections from the frozen units to units of the following layer or layers are not changed for the training run. Thus, there is no need to update weights of output connections from frozen units to units of following layer(s) during each training run. When units are frozen by the freeze-out component 108 as opposed to being removed when utilizing dropout, the architecture of the neural network remains unchanged. This eliminates a step of updating the weights of output connections from dropped units when utilizing dropout.” (Tan, [0034]). One subset of parameters is frozen for training, while the other is trained and updated as normal.
Tan relates to selective freezing of neural network parameters and is analogous to the claimed invention. The existing combination teaches a method of identifying and unpruning important parameters initially pruned but later found to be important during training. The claimed invention improves upon this method by freezing the active set of parameters while the pruned subset is trained. Tan teaches a method of freezing a subset of a neural network’s parameters, applicable to the existing combination. A person of ordinary skill in the art would have recognized that freezing the active parameters while evaluating pruned parameters would lead to the predictable result of only training the pruned parameters during this phase, and would improve the known device by increasing the efficiency of training at this phase by only training the parameters being evaluated for importance (the pruned parameters) (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
The analysis of claims 19-25 mirrors that of claims 1-2, 8-9, 12-13, and 15-17, with the exception that claims 19-25 are directed to generic computer hardware which executes the methods of claims 1-2, 8-9, 12-13, and 15-17. This generic hardware is taught by Yao, as discussed regarding claim 18. Thus, claims 19-25 are rejected under the same rationales used for claims 1-2, 8-9, 12-13, and 15-17, respectively.
Regarding claim 26, Ma discloses instructions, in an iteration of at least one iteration of a neural network model sparsification process, to:
train an active set of parameters in a neural network model from which a subset of parameters has been temporarily pruned:
“During model training, one of the sparse partitions is grown to dense while the rest remain sparse. After some epochs of training, the previously dense partition is pruned to sparse, and an alternate partition is grown to dense. This process is repeated (iterated) so that all partitions are grown to dense and pruned back to sparse multiple times” (Ma, page 2, paragraph 3)
PNG
media_image1.png
401
1162
media_image1.png
Greyscale
(Ma, page 2, Figure 1)
“As shown in Figure 1, the cyclic GaP (C-GaP) method rotates the dense partitions among all K partitions (K = 4 in Figure 1). Starting from all partitions with random sparse masks, the first partition is grown to a dense one. All weights in the first partition and the masked weights in the remaining 3 partitions are trained for a few epochs (Step 1). Then, the first partition is pruned to sparse (Step 2). Next, we apply the same strategy to the second partition, then continue iterating over all K partitions. When all layers are cyclically grown and pruned once after K steps, we call it one round. This process is repeated K times.” (Ma, page 3, paragraph 2). Each partition is grown and pruned K times (with prunings prior to the final round being temporary).
“In the scheduled GaP methods, all weights (parameters) to be grown (or to be pruned) belong to the same partition. All weights in the model are explored when every partition is grown to dense and pruned to sparse.” (Ma, page 2, paragraph 4)
estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model; and update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: “All of our experimental results are trained and inferenced using PyTorch in the machines with 8 NVIDIA-V100 GPUs. The cyclic GaP is trained in one training node and the parallel GaP is trained in K training nodes, where K is the number of model partitions. In the pruning stage of the GaP method, the weights with the lowest magnitudes (importance) are pruned to zero.” (Ma, page 5, paragraph 2). The active set is updated to include only connections with greater importance.
Ma relates to dynamic pruning methods for neural networks and is analogous to the claimed invention.
While Ma fails to disclose the further limitations of the claim Yao discloses [a] non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device, in an iteration of at least one iteration of a neural network model sparsification process, to: “The material disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM) (non-transitory memory storage)” (Yao, [0018]
Yao further discloses instructions to estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model; and update the active set of parameters in the neural network model, based on the importance of the subset of parameters pruned from the neural network model: “The compression discussed herein may include iterative pruning and splicing operations and parameter weight update operations. Such pruning operations (e.g., disconnecting an available connection at a particular iteration) may compress the DNN model by removing unimportant connections and such splicing operations (e.g., reconnecting previously disconnected available connections at a particular iteration) may provide recovery for pruned connections that are found to be important over the iterations (training)” (Yao, [0021]). Pruned parameters found to be important are added back to the active set.
Yao relates to dynamic pruning and splicing methods for neural networks and is analogous to the claimed invention.
Ma teaches a method of neural network sparsification. The claimed invention improves upon this method by storing it in the form of instructions on computer hardware. Yao teaches computer hardware for a neural network sparsification system, applicable to Ma. A person of ordinary skill in the art would have recognized that storing Ma’s method as computer instructions on Yao’s hardware would lead to the predictable result of the method being executable by a computing system, and would improve the known device by allowing it to be performed with real data (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ma to return high-importance pruned parameters to the active set of parameters, as disclosed by Ma. This combination of pruning and splicing can provide sparse neural networks with large compression rates without losing significant accuracy. See Yao, [0022].
While Yao fails to disclose the further limitations of the claim, Tan discloses instructions to estimate an importance of the subset of parameters temporarily pruned from the neural network model by freezing the active set of parameters in the neural network model and training the subset of parameters in the neural network model: “In certain embodiments, the freeze-out component 108 can freeze the selected subset of units of the neural network so that weights of output connections from the frozen subset of units will not be updated for a training run. For example, when the freeze-out component 108 can freeze the selected subset of units of the neural network, the connections of frozen units to units of the following layer or layers are not removed or changed. The output of frozen units is not included for a training run but the weights of output connections from the frozen units to units of the following layer or layers are not changed for the training run. Thus, there is no need to update weights of output connections from frozen units to units of following layer(s) during each training run. When units are frozen by the freeze-out component 108 as opposed to being removed when utilizing dropout, the architecture of the neural network remains unchanged. This eliminates a step of updating the weights of output connections from dropped units when utilizing dropout.” (Tan, [0034]). One subset of parameters is frozen for training, while the other is trained and updated as normal.
Tan relates to selective freezing of neural network parameters and is analogous to the claimed invention. The existing combination teaches a method of identifying and unpruning important parameters initially pruned but later found to be important during training. The claimed invention improves upon this method by freezing the active set of parameters while the pruned subset is trained. Tan teaches a method of freezing a subset of a neural network’s parameters, applicable to the existing combination. A person of ordinary skill in the art would have recognized that freezing the active parameters while evaluating pruned parameters would lead to the predictable result of only training the pruned parameters during this phase, and would improve the known device by increasing the efficiency of training at this phase by only training the parameters being evaluated for importance (the pruned parameters) (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Srinivas et al. (Cyclical Pruning for Sparse Neural Networks, published 2/2/2022, arXiv:2202.01290v1) discloses a method of cyclically pruning a sparse neural network
Wimmer et al. (Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey, published 5/17/2022, arXiv:2205.08099v1) discloses a survey of methods for pruning and / or freezing parameters in neural networks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron P Gormley whose telephone number is (571)272-1372. The examiner can normally be reached Monday - Friday 12:00 PM - 8:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AG/Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148