Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite mathematical concepts of solving equation. This judicial exception is not integrated into a practical application nor does it include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are extra-solution activity in combination with generic computer components used to implement the abstract idea, see the rejection below for further details.
Claims 1 and 7
Step 1: The claims recites a device and method, therefore, it falls into the statutory categories of a method and a device.
Step 2A Prong 1: The claim recites, inter alia:
Adaptively determine at least one parameter of the sparsity regularization; (This is a mathematical concept that is done by solving an equation. See para. [0028-0030] that teaches the sparsity regularization is TL1 regularization is calculated using equation 1 in para. [0028]. Also, a parameter can be calculated using equation 4 in the specification at para. [0043].)
Step 2A Prong 2:
This judicial exception is no integrated into a practical application. Aside from the limitations above, the claim recites:
a memory configured to store at least one data; and at least one processor configured to execute a network lightweight module, wherein, when executing the network lightweight module, the processor is configured to: (The above limitation amount to using generic computer components to implement the abstract idea, see MPEP 2106.05(f).)
perform learning on an input neural network based on sparsity regularization by using scaling factor values of batch normalization and a target pruning ratio; (This limitation is cited a high level of generality and results in using a neural network at tool to implement the abstract idea, see MPEP 2106.05(f).)
perform pruning on the learning result; and perform fine tuning on the pruning result. (Both of these limitations at cited a high of generality and result in using the model, the neural network, as a tool to implement the abstract idea, see MPEP 2106.05(f).)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere generic computer components used to perform the disclosed abstract idea above.
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements of “a memory configured to store at least one data; and at least one processor configured to execute a network lightweight module, wherein, when executing the network lightweight module, the processor is configured to:”; “perform learning on an input neural network based on sparsity regularization by using scaling factor values of batch normalization and a target pruning ratio;”; and “perform pruning on the learning result; and perform fine tuning on the pruning result.” Are cited a high level of generality and result in using generic computer components to implement the abstract idea.
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are generic computer functions that are implemented to perform the disclosed abstract idea above.
Claim 2
Step 2A Prong 1: The claim recites, inter alia:
calculate a task loss and a regularization loss; (Both of these are mathematical concepts that done by solving an equation. Para. [0037 and 0038] of instant specification teaches the equation for task loss is equation 2 and regularization loss is equation 3. )
PNG
media_image1.png
68
462
media_image1.png
Greyscale
PNG
media_image2.png
70
450
media_image2.png
Greyscale
Step 2A Prong 2:
This judicial exception is no integrated into a practical application. Aside from the limitations above, the claim recites:
perform backpropagation based on the calculation result; and perform the learning based on the backpropagation result. (Both of these limitations at cited a high of generality and result in “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level application of learning and performing backpropagation based a result.)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere generic computer components used to perform the disclosed abstract idea above.
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements of “perform backpropagation based on the calculation result; and perform the learning based on the backpropagation result.” are cited a high level of generality and result in using generic computer components to implement the abstract idea.
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are generic computer functions that are implemented to perform the disclosed abstract idea above.
3. The device of claim 2, wherein the sparsity regularization is transformed L1 (TL1) regularization, and
PNG
media_image3.png
36
170
media_image3.png
Greyscale
wherein the TL1 regularization is expressed as
(This is a mathematical concept of solving an equation.)
4. The device of claim 3, wherein the task loss is expressed as Σx,yI(f(x,W),y), and
wherein the regularization loss is expressed as λΣγg(γ). (This is a mathematical concept of solving an equation.)
5. The device of claim 4, wherein the processor performs the learning by adaptively determining a parameter ‘a’. (The adaptively determining parameter a is a mathematical concept wherein an equation is solved, see para. [0043] and equation 4 for finding parameter a. Also using a processor to perform the calculation is using generic computer component to implement the abstract idea, see MPEP 2106.05(f).)
6. The device of claim 5, wherein the processor determines the parameter ‘a’ based on a learning batch ‘x’, a scaling factor ‘γ’ of the batch normalization, and a target pruning ratio ‘p’. (The adaptively determining parameter a is a mathematical concept wherein an equation is solved, see para. [0043] and equation 4 for finding parameter a. Also using a processor to perform the calculation is using generic computer component to implement the abstract idea, see MPEP 2106.05(f).)
Claim 8
Step 2A Prong 1: The claim recites, inter alia:
calculate a task loss and a regularization loss; and calculate a total loss. (These are mathematical concepts that done by solving an equation. Para. [0037 and 0038] of instant specification teaches the equation for task loss is equation 2, regularization loss is equation 3, and para. [0040] teaches total loss calculated by adding task loss and regularization loss. )
PNG
media_image1.png
68
462
media_image1.png
Greyscale
PNG
media_image2.png
70
450
media_image2.png
Greyscale
Step 2A Prong 2:
This judicial exception is no integrated into a practical application. Aside from the limitations above, the claim recites:
perform backpropagation after calculating a total loss from the calculated loss and calculated regularization loss. (This limitations at cited a high of generality and result in “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level application performing backpropagation.)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere generic computer components used to perform the disclosed abstract idea above.
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional element of “perform backpropagation after calculating a total loss from the calculated loss and calculated regularization loss” are cited a high level of generality and result in using generic computer components to implement the abstract idea.
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are generic computer functions that are implemented to perform the disclosed abstract idea above.
Claim 9
PNG
media_image3.png
36
170
media_image3.png
Greyscale
9. The method of claim 8, wherein the sparsity regularization is transformed L1 (TL1) regularization, and
wherein the TL1 regularization is expressed as (This is a mathematical concept of solving an equation.)
Claim 10
10. The method of claim 9, wherein the task loss is expressed as Σx,yI(f(x,W),y). (This is a mathematical concept of solving an equation.)
Claim 11
11. The method of claim 10, wherein the wherein the regularization loss is expressed as λΣγg(γ). (This is a mathematical concept of solving an equation.)
Claim 12
12. The method of claim 11, wherein the adaptively determining of the at least one parameter of the sparsity regularization includes: adaptively determining a parameter ‘a’. (The adaptively determining parameter a is a mathematical concept wherein an equation is solved, see para. [0043] and equation 4 for finding parameter a. Also using a processor to perform the calculation is using generic computer component to implement the abstract idea, see MPEP 2106.05(f).)
Claim 13
Step 2A Prong 1: The claim recites, inter alia:
Sorting the scaling factor ‘y’; (This is a mental processing of sorting data based on values, it can be done with the aid of pen and paper.)
Assigning a parameter ‘th’ by calculating a value corresponding to the target pruning ration ‘p’ in the sorted scaling factor ‘y’; and calculating the parameter “a” from the assigned parameter “th”. (Both of these are mathematical concepts that done by solving an equation. Para. [0042-0044] of instant specification teaches the computer system calculates the value corresponding to ‘th’ based on equation 4. Para. [0042-0044] cites
PNG
media_image4.png
414
524
media_image4.png
Greyscale
)
Step 2A Prong 2:
This judicial exception is no integrated into a practical application. Aside from the limitations above, the claim recites:
receiving a learning batch ‘x’, a scaling factor ‘γ’ of the batch normalization, and a target pruning ratio ‘p’; (This amount to data gathering which is extra-solution activity, see MPEP 2106.05(g).)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere extra-solution activity in combination with that abstract idea.
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements of “receiving a learning batch ‘x’, a scaling factor ‘γ’ of the batch normalization, and a target pruning ratio ‘p’;” are well-understood, routine and conventional. See MPEP 2106.06(d)(II) wherein it cites “The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. i. Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are well-understood, routine and conventional extra-solution activity in combination with the abstract idea.
Claim 14
14. The method of claim 13, wherein the calculating of the parameter ‘a’ from the assigned parameter ‘th’ satisfies a condition of
PNG
media_image5.png
44
116
media_image5.png
Greyscale
. (This is a mathematical concept of solving an equation.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2 and 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over by Phan et al. (US 2020/0293876 A1 – hereinafter Phan) and in further view of Liu et al. (“Learning Efficient Convolutional Networks through Network Slimming” – hereinafter Liu).
In regards to claim 1, Phan discloses a deep neural network lightweight device based on batch normalization, the device comprising:
a memory configured to store at least one data; and (Phan fig. 5 element 506 teaches memory for storing data.)
at least one processor configured to execute a network lightweight module, (Phan fig. 5 element 504 teaches a processor for executing a model, and para. [0012] teaches the DNN is designed for low-resource environments such as mobile devices and IoT edge devices, thus it a lightweight model.))
wherein, when executing the network lightweight module, the processor is configured to:
perform learning on an input neural network based on sparsity regularization to adaptively determine at least one parameter of the sparsity regularization and using a target pruning ratio; ( Phan para. [0027] teaches “In this embodiment, neural network compression program 132 uses a L1-regularized model with a convex objective to solve for the new sparse weights {Ŵ1 . . . ŴL}, calculated using model (1) for some positive parameter {α>0}.” This teaches learning based on sparsity regularization (L1 regularization) to determine at least one parameter. See para. [0030-0032 and 0039] wherein parameters are updated at each step as equations are solved iteratively. Phan teaches a pruning ration as a compression ratio that controls the number of zero and nonzero weighted connections of the neural network. See Phan para. [0025] which cites “Compression ratio is defined as a ratio of the number of zero weights over the number of nonzero weights in a resulting compressed neural network.”, para. [0028] teaches the Lo constrained model where the maximum allowed number of nonzero, K, is determined based on the compression ratio. The examiner maps this the pruning ration as it weights with zero is effectively pruned or not connected.)
perform pruning on the learning result; and (Phan para. [0011] teaches pruning connections of the DNN based on the optimization results, wherein it cites “Embodiments of the present invention provide a compression method that prunes many connections from a pre-trained DNN while maintaining comparable accuracy. The compression method is based on solving tractable optimization problems for pruning DNNs. More specifically, embodiments of the present invention provide a tractable optimization model to compress a DNN by pruning nodes and weights while giving a high compression rate and maintaining accuracy. This optimization model: (1) automatically prunes redundant connections layer-by-layer in a pre-trained DNN;
perform fine tuning on the pruning result.”)
perform fine tuning on the pruning result. (Phan para. [0044] teaches fine tuning the neural network wherein it cites “In step 240, neural network compression program 132 re-trains and fine-tunes the neural network. In an embodiment, neural network compression program 132 re-trains and fine-tunes neural network 112 with the new sparse weights {Ŵ1 . . . ŴL} between each layer.”)
However, Phan does not explicitly disclose using scaling factor values of batch normalization.
Liu discloses using scaling factor values of batch normalization. (Liu discloses a system that uses scaling factor values of batch normalization. (Liu page 2 fig. 1 cites “We associate a scaling factor (reused from a batch normalization layer) with each channel in convolutional layers. Sparsity regularization is imposed on these scaling factors during training to automatically identify unimportant channels. The channels with small scaling factor values (in orange color) will be pruned (left side). After pruning, we obtain compact models (right side), which are then fine-tuned to achieve comparable (or even higher) accuracy as normally trained full network.”, also see page 4 section “Channel Pruning and Fine-Tuning” cites “We prune channels with a global threshold across all layers, which is defined as a certain percentile of all the scaling factor values. For instance, we prune 70% channels with lower scaling factors by choosing the percentile threshold as 70%. By doing so, we obtain a more compact network with less parameters and run-time memory, as well as less computing operations.” Thus, it teaches using scaling factor values of batch normalization and using a pruning ratio.)
It would have been obvious to one of ordinary skill in the art before earliest effective filing date of the claimed invention to modify the teachings of the Phan with that Liu in order to allow for using scaling factor values of batch normalization when pruning connections as both reference deal with slimming a network by pruning connections of the network and it provides the benefit of obtaining a more compact network with less parameters and run-time memory, as well as less computing operations as stated in Liu on page 4 right column second paragraph.
In regards to claim 2, Phan in view of Liu disclose the device of claim 1, wherein the processor is configured to: calculate a task loss and a regularization loss; (Phan para. [0027-0031] teaches a first optimization model that builds optimization models between layers so that difference between the pre-trained output X and new output X’ is “calculated using a mean square error loss function”, is minimized, and it adds an L1 regularization penalty, which is regularization loss, see eq. 1 after para. [0027]. These paragraph further teaches L0 sparsity constraint terms and an augmented Lagrangian/penalty for the compression optimization.) perform backpropagation based on the calculation result and perform the learning based on the backpropagation result.(Phan para. [0030] teaches stochastic gradient descent wherein it cites “variance-reduction stochastic gradient descent. At each iteration, a sample point is randomly picked, the gradient for the associated loss function is computed, and the decision variable is updated. For each update, the gradient step is projected into the sparsity constraint. After a certain number of inner iterations, a full gradient is calculated to reduce the gradient variance.”, also see Phan para. [0044]. This teaches backpropagation and learning from it.)
In regards to claim 7, it is the method embodiment of claim 1 with similar limitation and thus rejected using the same reasoning found in claim 1.
In regards to claim 8, it is the method embodiment of claim 2 with similar limitation and thus rejected using the same reasoning found in claim 2.
Claims 3-6 and 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over by Phan et al. (US 2020/0293876 A1 – hereinafter Phan) in view of Liu et al. (“Learning Efficient Convolutional Networks through Network Slimming” – hereinafter Liu) and in further view of Ma et al. - (“Transformed L1 Regularization for Learning Sparse Deep Neural Networks” ).
In regards to claim 3, Phan in view of Liu discloses the device of claim 2, but fails to explicitly discloses wherein the sparsity regularization is transformed L1 (TL1) regularization, and wherein the TL1 regularization is expressed as:
PNG
media_image6.png
70
184
media_image6.png
Greyscale
Ma et al. discloses sparsity regularization is transformed L1 (TL1) regularization, and wherein the TL1 regularization is expressed as:
PNG
media_image6.png
70
184
media_image6.png
Greyscale
(Ma et al. page 288 equation 7 discloses the transformed L1 regularization equation wherein it cites the transformed L1 regularization equation, also see question 8 wherein the summation is added.
PNG
media_image7.png
174
687
media_image7.png
Greyscale
)
It would have been obvious to one of ordinary skill in the art before the earliest effective filing date of the claimed invention to modify the teachings of Phan in view of Liu with that of Ma et al. in order to allow for using transformed L1 regularization as all the references deal with using L1 regularization, and the benefit of using the transformed L1 regularization is that it provides superior sparsity handling and reduces estimation bias when compared the standard L1 regularization.
In regards to claim 4, Phan in view of Liu in view of Ma et al. disclose the device of claim 3, wherein the task loss is expressed as
PNG
media_image8.png
32
119
media_image8.png
Greyscale
and wherein the regularization loss is expressed as
PNG
media_image9.png
31
80
media_image9.png
Greyscale
. (Liu page 3 equation 1 in “Scaling Factors and Sparsity-Induced Penalty” discloses the following: “
PNG
media_image10.png
276
332
media_image10.png
Greyscale
).
In regards to claim 5, Phan in view of Liu in view of Ma et al. disclose the device of claim 4, wherein the processor performs the learning by adaptively determining a parameter 'a'. (See Ma equation 4 and section 3.1 that teaches changing and find the parameter “a”.)
In regards to claim 6, Phan in view of Liu in view of Ma et al. disclose the device of claim 5, wherein the processor determines the parameter 'a' based on a learning batch 'x', a scaling factor 'y' of the batch normalization, and a target pruning ratio 'p'. (Liu fig. 1 cites “We associate a scaling factor (reused from a batch normalization layer) with each channel in convolutional layers. Sparsity regularization is imposed on these scaling factors during training to automatically identify unimportant channels. The channels with small scaling factor values (in orange color) will be pruned (left side). After pruning, we obtain compact models (right side), which are then fine-tuned to achieve comparable (or even higher) accuracy as normally trained full network.”; Liu page 3 section “Scaling Factors and Sparsity-Induced Penalty” section cites “Specifically, the training objective of our approach is given by equation (1) where (x; y) denote the train input and target, W denotes the trainable weights, the first sum-term corresponds to the normal training loss of a CNN, g(∙) is a sparsity-induced penalty on the scaling factors, and λ balances the two terms.”; Liu page 4 “Channel Pruning and Fine-tuning” section cites “We prune channels with a global threshold across all layers, which is defined as a certain percentile of all the scaling factor values.” So, Liu disclose the learning batch, batch normalization scaling factors and target pruning ratio wherein a model trains with a regularizer on BN scaling factors and then uses the percentile threshold (pruning ratio) to prune channels. Then Ma equation 4 and section 3.1 that teaches changing and find the parameter “a”.)
In regards to claim 9, it is the method embodiment of claim 3 with similar limitation and thus rejected using the same reasoning found in claim 3.
In regards to claim 10, it is the method embodiment of claim 4 with similar limitation and thus rejected using the same reasoning found in claim 4.
In regards to claim 11, it is the method embodiment of claim 4 with similar limitation and thus rejected using the same reasoning found in claim 4.
In regards to claim 12, it is the method embodiment of claim 5 with similar limitation and thus rejected using the same reasoning found in claim 5.
Response to Arguments
Applicant's arguments filed 17 December 2025 have been fully considered but they are not persuasive. The applicant argues that the rejection of the claims under 35 USC 101 for being abstract idea is improper as the claims improve the technology by covering a particular solution to a problem. Applicant argues that adaptively determining one parameter of sparsity regularization by using scaling factor values of batch normalization and a target pruning ratio allow a lightweight device the recognition ability of a neural network. The examiner respectfully traverses the applicant’s arguments as the improvements themselves are abstract ideas of performing mathematically calculations by solving equations. See para. [0028-0030] that teaches the sparsity regularization is TL1 regularization is calculated using equation 1 in para. [0028]. Also, a parameter can be calculated using equation 4 in the specification at para. [0043]. Thus, this claims are not an improvement to technology but rather an abstract idea itself.
In regards to applicant argument as it relates to the rejection of claims 1 and 7 under 35 USC 102, applicant’s arguments with respect to claims 1-14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Allowable Subject Matter
Claims 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULINHO E SMITH whose telephone number is (571)270-1358. The examiner can normally be reached Mon-Fri. 10AM-6PM CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at 571-270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULINHO E SMITH/Primary Examiner, Art Unit 2127
Prosecution Insights