Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to the 101 rejection of claims 1, 3-9, 11-16 and 18-23 are persuasive.
Applicant argues,
Han thus fails to disclose that "when an updated target weight corresponding to the target block is not comprised in the N largest target weights in the M updated target weights, the second block in the first neural network is further used to perform, based on an output of the target block, the operation corresponding to the second block" as recited in claim 1. Though this feature, after updating a connection relationship, a connection between the target block and the second block is reserved.
Remarks 16.
Applicants blocks are collections of nodes, and they are taught by the layer(s) of Han, see fig. 3.
PNG
media_image1.png
348
284
media_image1.png
Greyscale
PNG
media_image2.png
348
284
media_image2.png
Greyscale
This reading that makes sense given Applicant’s semi-serial connections shown in figures 4 and 5 below, and the general state of the art where many nodes make up a layer and a layer is often serially connected to the layer before it and after it.
PNG
media_image3.png
336
642
media_image3.png
Greyscale
When Applicant claims “block, the operations corresponding to the first blocks, wherein N is an integer that is less than M, the M first blocks in the initial neural network comprise a target block, the target block is connected to the second block on the serial connection, and when an updated target weight corresponding to the target block is not comprised in the N largest target weights of the M updated target weights, the target block in the first neural network is further used to perform, based on the output of the second block, the operation corresponding to the target block.” Claim 9. Said another way, each layer has M first blocks, those M first blocks are connected directly/serially to the target block. The N largest-weight blocks of the M blocks have their connection with the target block reserved, the rest are pruned. This is taught by Learning both Weights and Connections for Efficient Neural Networks by Han Fig. 3 and page 2, “prune network connections in a manner that preserves the original accuracy. After an initial training phase, we remove all connections whose weight is lower than a threshold.”
PNG
media_image4.png
338
596
media_image4.png
Greyscale
The order of the first blocks and second block is reversed in claim 1, but it is the same concept. Therefore the claims are taught by Han.
Allowable Subject Matter
Claims 21-23 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art of record does not teach or make obvious the idea of reserving the N largest-weight connections and also reserving the smallest-weight n+1 connections. The art doesn’t teach this, because there is no reason to make a determination about the weight of a connection if you know that you’re going to keep the connection regardless. The prior art just keeps the connection and skips the computation.
Claim Rejections - 35 USC § 101
All 101 rejections withdrawn.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 3, 6-9, 11, 14-16 and 20 are rejected under 35 U.S.C. 102(a)(1) as being described by Learning both Weights and Connections for Efficient Neural Networks by Han et al.
Han teaches claims 1, 9 and 16. (Currently Amended) A neural network structure determining method comprising: (Han title “Learning both Weights and Connections for Efficient Neural Networks”)
obtaining a to-be-trained initial neural network, wherein the initial neural network comprises M first blocks and a second block, the second block is connected to each of the first blocks, each of the first blocks corresponds to a respective one of target weights, the second block is used to perform, based on M first outputs, an operation corresponding to the second block, the M first outputs each are obtained by performing a product operation on an output of each of the first blocks and the respective corresponding target weights, and the each of the target weights is a trainable weight, wherein M is an integer greater than 1, the second block and the M first blocks in the initial neural network sequentially form a serial connection, and the second block is a start point of the serial connection; (Han fig. 3 shows, see below, the initial network on the left. The second block is any one of the nodes in a downstream layer. The first M blocks are the nodes preceding layer. Each node performs a summation of the inputs and multiplies the sum by the weight on the downstream connection of the node. This order of second block and first blocks is reversed in claim 1, but it is still taught by fig. 3.)
PNG
media_image4.png
338
596
media_image4.png
Greyscale
performing model training on the initial neural network, to obtain M updated target weights; and (Han fig. 2 “train weights”, see below. Training weights updates weights to a target weight.)
PNG
media_image5.png
188
228
media_image5.png
Greyscale
updating a connection relationship between the second block and the M first blocks in the initial neural network based on the M updated target weights, to obtain a first neural network, wherein the second block in the first neural network is used to perform, based on outputs of N first blocks corresponding to N largest target weights of the M updated target weights, the operation corresponding to the second block, wherein N is an integer that is less than M, the M first blocks in the initial neural network comprise a target block, the target block is connected to the second block on the serial connection, and when an updated target weight corresponding to the target block is not comprised in the N largest target weights of the M updated target weights, the target block in the first neural network is further used to perform, based on the output of the second block, the operation corresponding to the target block. (Han fig. 2 above “prune connections”. Han Fig. 3 shows an updated connection relationship on the right side, where the connections are pruned. Han p. 2 “prune network connections in a manner that preserves the original accuracy. After an initial training phase, we remove all connections whose weight is lower than a threshold.”)
Han teaches claims 3 and 11. (Original) The method according to claim 2, wherein N is 1. (Han fig. 3 shows an example of an embodiment where N can be 1, see below highlighted connection.)
PNG
media_image6.png
534
432
media_image6.png
Greyscale
Han teaches claims 6, 14 and 20. (Original) The method according to claim 1, wherein quantities of input channels and output channels of each of the M first blocks are the same as quantities of input channels and output channels of the second block. (Han fig. 3 below shows one node with the same inputs and outputs. Because the pruning is iterative each new pruned network can be though of as having M blocks before it is pruned again.)
PNG
media_image7.png
534
432
media_image7.png
Greyscale
Han teaches claims 7 and 15. (Currently Amended) The method according to claim 1, wherein
the method further comprises calculating a summation result of the M first outputs; and (Han fig. 3 shows lines converging at an input of a node. That convergence is summation.)
the second block in the initial neural network is used to perform, based on the summation result of the M first outputs, the operation corresponding to the second block; (Han fig. 3 shows lines converging at an input of a node. That convergence is summation, that sum is later multiplied by the weight of the next connections.)
the method further comprises calculating a summation result of the outputs of the first blocks corresponding to the N largest target weights of the M updated target weights; and (Han fig. 3 shows a pruned network, it is the same principle though, The inputs are summed, the output of the node is multiplied by a weight.)
the second block in the first neural network is used to perform, based on the summation result of the outputs of the first blocks corresponding to the N largest target weights of the M updated target weights, the operation corresponding to the second block. (Han p. 2 “prune network connections in a manner that preserves the original accuracy. After an initial training phase, we remove all connections whose weight is lower than a threshold.”)
Han teaches claim 8. (Currently Amended) The method according to claim 1, wherein the method further comprises:
obtaining to-be-trained data, wherein the to-be-trained data comprises: image data, text data, and/or voice data; and correspondingly, the performing of the model training on the initial neural network comprises:
performing model training on the initial neural network based on the to-be-trained data. (Han abs “On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9×…”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4, 5, 12, 13, 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Learning both Weights and Connections for Efficient Neural Networks by Han et al and USPBPUB 20220164671 to Dong et al.
Han teaches claims 4, 12 and 18. (Currently Amended) The method according to claim 1, wherein the performing of the model training on the initial neural network, to obtain the M updated target weights comprises:
performing model training on the initial neural network (Han p. 6 sec. 4.3 “We used five iterations of pruning an [sic] retraining.” Retraining networks necessitates a first training, and training updates weights.) Han doesn’t disclose the number of iterations inside of training cycle.
However, Dong teaches training on the initial neural network for a first preset quantity of iterations, (Dong para 88 “there may be a situation where the retraining process does not converge. That is, the above change cannot converge within the preset range after multiple iterations. Thus, a maximum number of iterations can be preset.”)
Dong, Han and the claims all train and prune neural networks.1 It would have been obvious to a person having ordinary skill in the art, at the time of filing, to put a cap on the number of training iterations because it is wasteful for the model not to converge and the preset number of iterations caps the amount of wasted compute caused by a model that doesn’t converge. Dong para 88.
Han teaches claims 5, 13 and 19. (Original) The method according to claim 1, wherein the method further comprises:
performing model training on the first neural network until data processing precision of the first neural network (Han p. 6 sec. 4.3 “We used five iterations of pruning an [sic] retraining.” Each retrain creates a new NN, so the first retrain would create Applicant’s second neural network.) Han doesn’t teach preset iterations.
However, Dong teaches meets a preset condition or a quantity of iterations of model training reaches a second preset quantity of iterations… (Dong para 95 “If the number of iterations reaches the preset number M, … if the number of iterations doesn't reach the preset number M, the method goes back block 160 and continues to iteratively performing the forward propagation and back propagation blocks…. For example, the preset number M ranges from 3 to 20. More specifically, in some examples, the preset number M may be 3, 5 or 10.” To be clear, Han teaches retraining. Dong just teaches a variety of iteration caps inside of training.)
Dong, Han and the claims all train and prune neural networks.2 It would have been obvious to a person having ordinary skill in the art, at the time of filing, to put a cap on the number of training iterations because it is wasteful for the model not to converge and the preset number of iterations caps the amount of wasted compute caused by a model that doesn’t converge. Dong para 88.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
1 Dong para 21 “neural network compressed by method of the present application at different pruning ratios.”
2 Dong para 21 “neural network compressed by method of the present application at different pruning ratios.”
Prosecution Insights