DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are presented for examination.
Continued Examination under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on June 8, 2026 has been entered.
Response to Amendment
Applicant’s amendment has obviated the remaining objections to the drawings and specification (but has necessitated a new ground of objection, see below). Therefore, those objections are withdrawn.
Specification
The abstract of the disclosure does not commence on a separate sheet in accordance with 37 CFR 1.52(b)(4) and 1.72(b). A new abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-5, 7-12, and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Shen et al. (US 20220292360) (“Shen”) in view of Zhuo et al. (US 11030528) (“Zhuo”) and further in view of Chang et al. (US 20210374562) (“Chang”).
Regarding claim 1, Shen discloses “[a] processor, comprising:
one or more circuits to:
compute initial scores of one or more portions of one or more neural networks (process for a system of neural network pruning involves determining a sub-network [i.e., less than all portions of the neural networks] and calculating an early pruning indicator (EPI) value [initial score]; system determines a sub-network by ranking neurons based on calculated importance scores for each neuron; system then determines whether EPI is greater than a threshold and EPI from past epochs [previously evaluated portions of the networks]; the system then sets a status to prune and neurons are pruned such that only the top k neurons of the neural network remain – Shen, paragraphs 125-30 and Figs. 7-8) …;
compare the initial scores with scores of one or more previously evaluated portions of the one or more neural networks to identify a subset of the previously evaluated portions to use (system then determines whether EPI is greater than a threshold and EPI from past epochs [previously evaluated portions of the networks]; the system then sets a status to prune and neurons are pruned such that only the top k neurons [subset of the previously evaluated portions] of the neural network remain – Shen, paragraphs 125-30 and Figs. 7-8 ) …;
deactivate at least one portion of the one or more portions of the one or more neural networks identified according to the … scores compared with a pruning threshold (system then determines whether EPI is greater than a threshold [pruning threshold] and EPI [score] from past epochs; the system then sets a status to prune and neurons are pruned [deactivated] such that only the top k neurons of the neural network remain – Shen, paragraphs 125-30 and Figs. 7-8); and
perform one or more inferencing tasks with the one or more neural networks with at least the one portion being deactivated (if no epochs remain, a neural network is returned and utilized to perform various processes, such as image classification, object detection, segmentation, data analysis, and/or similar processes [inferencing tasks] – Shen, paragraph 133 and Figs. 7-8).”
Shen appears not to disclose explicitly the further limitations of the claim. However, Zhuo discloses “comput[ing] … scores of one or more portions of one or more neural networks based, at least in part, on respective output of the one or more portions (the highest accuracy of verification sets before and after pruning is compared, and if the highest accuracy of the verification set after pruning is greater than or equal to the highest accuracy of the verification set [output] before pruning, the current pruned ratio [score] is taken as a new lower limit of the pruned ratio and increased – Zhuo, claim 1)… [and]
modify[ing] the initial scores (he highest accuracy of verification sets before and after pruning is compared, and if the highest accuracy of the verification set after pruning is greater than or equal to the highest accuracy of the verification set [output] before pruning, the current pruned ratio [score] is taken as a new lower limit of the pruned ratio and increased [modified] – Zhuo, claim 1) ….”
Zhuo and the instant application both relate to pruning neural networks and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shen to calculate a pruning score based on output of the networks and then modify the score, as disclosed by Zhuo, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would speed up the computation of the network and reduce hardware requirements. See Zhuo. Col. 1, l. 65-col. 2, l. 2.
Neither Shen nor Zhuo appears to disclose explicitly the further limitations of the claim. However, Chang discloses that “the initial scores indicate respective importance of the one or more portions to a capability of the one or more neural networks (Chang paragraph 17 discloses that an importance score for each feature [portion] is inputted into a baseline version of a machine learning model and that the importance scores represent the aggregated impact of a feature on rankings [capability] outputted by the baseline version of the machine learning model over a period of time); … [and]
modify[ing] the initial scores to indicate changed respective importance (rank-based overlaps are calculated between a set of original rankings and a corresponding set of modified rankings for a given modified feature, and an analysis apparatus aggregates the risk-based overlaps into an importance score for the modified feature [modified score]; analysis apparatus calculates a similarity score of a feature as the average rank-based overlap between a set of modified rankings and a corresponding set of original rankings [where non-overlap indicates changed importance] – Chang, paragraph 50) ….”
Chang and the instant application both relate to simplification of machine learning models and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Shen and Zhuo to modify initial importance scores to reflect changed importance, as disclosed by Chang, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would improve resource consumption, latency, and/or scalability of the models by allowing the system to determine which features are unnecessary. See Chang, paragraph 6.
Claim 8 is a system claim corresponding to processor claim 1 and is rejected for the same reasons as given in the rejection of that claim. Similarly, claim 15 is a non-transitory machine-readable medium claim corresponding to processor claim 1 and is rejected for the same reasons as given in the rejection of that claim.
Regarding claim 2, Shen/Zhuo/Chang discloses that “the one or more portions of the one or more neural networks include one or more first neurons of a layer of the one or more neural networks, and the one or more previously evaluated portions of the one or more neural networks include one or more second neurons of one or more previously evaluated layers of the one or more neural networks (if a system for neural network pruning determines that a calculated early pruning indicator value is greater than or equal to a stability threshold and is greater than or equal to calculated early pruning indicator values for one or more past epochs [previous evaluations], the system indicates that the neural network is to be pruned – Shen, paragraph 89; sub-network of a first neural network is formed by one or more neurons per layer of a first neural network – id. at paragraph 75; difference between sub-networks for an lth layer may be defined – id. at paragraph 86 [i.e., the system evaluates some neurons in some layers in one epoch and other neurons in other layers in other epochs]).”
Claim 9 is a system claim corresponding to processor claim 2 and is rejected for the same reasons as given in the rejection of that claim. Similarly, claim 16 is a non-transitory machine-readable medium claim corresponding to processor claim 2 and is rejected for the same reasons as given in the rejection of that claim.
Regarding claim 3, Shen discloses that “the deactivation of the one or more portions of the one or more neural networks comprises removing the one or more portions based, at least in part, on the pruning threshold and the … initial scores representing importance of the one or more portions within the one or more neural networks (process for a system of neural network pruning involves determining a sub-network and calculating an early pruning indicator (EPI) value; system determines a sub-network by ranking neurons based on calculated importance scores for each neuron [portion]; system then determines whether EPI is greater than a threshold and EPI from past epochs; the system then sets a status to prune and neurons are pruned [deactivated] such that only the top k neurons of the neural network remain – Shen, paragraphs 125-30 and Figs. 7-8).”
Zhuo discloses “modified scores,” as shown in the rejection of claim 1. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shen/Chang to modify the pruning score, as disclosed by Zhuo, for substantially the same reasons as given in the rejection of claim 1.
Claim 10 is a system claim corresponding to processor claim 3 and is rejected for the same reasons as given in the rejection of that claim. Similarly, claim 17 is a non-transitory machine-readable medium claim corresponding to processor claim 3 and is rejected for the same reasons as given in the rejection of that claim.
Regarding claim 4, Shen discloses that “the pruning threshold is set based, at least in part, on ranking the … initial scores of the one or more portions within the one or more neural networks (system determines a sub-network by ranking neurons based at least in part on calculated importance scores for said neurons , and the system calculates an EPI value for the sub-network; the system then determines whether the EPI is greater than a threshold – Shen, paragraphs 125-26; grid search is utilized to determine the stability threshold – id. at paragraph 112; grid search analyzes every neuron during one or more epochs to determine a most optimal set of neurons to remove – id. at paragraph 107 [i.e., the grid search used to determine the threshold determines which neurons to remove, meaning that it is based on the ranking that determines which neurons to remove]).”
Zhuo discloses “modified scores,” as shown in the rejection of claim 1. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shen/Chang to modify the pruning score, as disclosed by Zhuo, for substantially the same reasons as given in the rejection of claim 1.
Claim 11 is a system claim corresponding to processor claim 4 and is rejected for the same reasons as given in the rejection of that claim. Similarly, claim 18 is a non-transitory machine-readable medium claim corresponding to processor claim 4 and is rejected for the same reasons as given in the rejection of that claim.
Regarding claim 5, Shen/Zhuo/Chang discloses that “the one or more circuits are further to:
calculate one or more first metrics associated with the one or more first neurons of the layer (system determines a sub-network by ranking [calculating metrics on] neurons based on calculated importance scores for each neuron – Shen, paragraph 125);
calculate one or more second metrics associated with the one or more second neurons of the one or more previously evaluated layers (system determines a sub-network by ranking neurons based on calculated importance scores for each neuron; system then determines whether EPI is greater than a threshold and EPI from past epochs [previously evaluated portions of the networks] – Shen, paragraphs 125-30; sub-network of a first neural network is formed by one or more neurons per layer of a first neural network – id. at paragraph 75; difference between sub-networks for an lth layer may be defined – id. at paragraph 86 [i.e., the system evaluates some neurons in some layers in one epoch and other neurons in other layers in other epochs]); and
deactivate the one or more first neurons based, at least in part, on the one or more first metrics and the one or more second metrics (if EPI is greater than EPI values from past epochs, the system sets a status to prune [deactivate neurons] – Shen, paragraphs 126-27; see also paragraph 125 (disclosing that the system calculates the EPI value based on the sub-network formed by the ranking [i.e., based on the metrics])).”
Claim 12 is a system claim corresponding to processor claim 5 and is rejected for the same reasons as given in the rejection of that claim. Similarly, claim 19 is a non-transitory machine-readable medium claim corresponding to processor claim 4 and is rejected for the same reasons as given in the rejection of that claim.
Regarding claim 7, Shen/Zhuo/Chang discloses that “the one or more first metrics and the one or more second metrics are based, at least in part, on an L2-norm (magnitude-based criterion refers to a criterion to rank neurons that uses an l2-norm of neuron weights to measure a relevance of a neuron in a network – Shen, paragraph 70).”
Claim 14 is a system claim corresponding to processor claim 7 and is rejected for the same reasons as given in the rejection of that claim.
Claims 6, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shen in view of Zhuo and Chang and further in view of Miret et al. (US 20220092425) (“Miret”).
Regarding claim 6, Shen/Zhuo/Chang appears not to disclose explicitly the further limitations of the claim. However, Miret discloses that “to compare the initial scores with the scores of the one or more previously evaluated portions, the one or more circuits are to determine a sum of a portion of the one or more second metrics having higher values than the one or more first metrics (pruning module selects a subset of the filters based on the pruning ratio; for instance, where the pruning ratio is 10%, the filter pruning module selects 10% of the filters based on the ranking, e.g., the 10% filters that have lower absolute magnitude sum than the remaining 90% filters [second metric = absolute magnitude of 90% of filters; first metric = sum of absolute magnitudes of 10% of filters] – Miret, paragraph 71).”
Miret and the instant application both relate to pruning of neural networks and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shen/Zhuo/Chang to perform the pruning based on a ratio of a metric related to lower-performing neurons to that of higher-performing neurons, as disclosed by Miret, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would increase the sparsity in the hidden layers, thereby reducing the memory footprint and the processor resources consumed in executing the model. See Miret, paragraph 71.
Claim 13 is a system claim corresponding to processor claim 6 and is rejected for the same reasons as given in the rejection of that claim. Similarly, claim 20 is a non-transitory machine-readable medium claim corresponding to processor claim 6 and is rejected for the same reasons as given in the rejection of that claim.
Response to Arguments
Applicant’s arguments with respect to the claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN C VAUGHN whose telephone number is (571)272-4849. The examiner can normally be reached M-R 7:00a-5:00p ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at 571-272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN C VAUGHN/ Primary Examiner, Art Unit 2125
Prosecution Insights