Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
Claim Rejections – 35 U.S.C. 101
Applicant’s amendments have been fully considered but they are not persuasive.
Applicant argues (pg. 8-9) that “The present claims provide a technical improvement in computer technology by providing a computationally efficient technique for training a sparse neural network that enables reduced consumption of computational resources during training and allows for the deployment of efficient neural network that take up less memory and require less compute at inference time. See Specification, paragraphs 46-47. The claimed invention thus provides a technical improvement to computer technology by improving the computational efficiency of machine learning training.”
Examiner respectfully disagrees. While paragraphs 46-47 of the Specification do appear to disclose the idea of “enables reduced consumption of computational resources during training and allows for the deployment of efficient neural network that take up less memory and require less compute at inference time”, the claim does not reflect this improvement to the technology. Examiner suggests amending the language of the claims to better reflect this idea.
Claim Rejections – 35 U.S.C. 103
Applicant’s amendments have been fully considered but they are not persuasive.
Applicant argues (pg. 2-4) that Evci and Xu do not address the following argument: “Applicant submits that amended claim 1 clarifies that the first and second sub-sets of network parameters are distinct: ‘the first sub-set of parameters comprises a subset of a largest of the plurality of network parameters’, and that ‘the second sub-set of parameters comprises a subset of next largest of the plurality of network parameters.’ Thus, "updating parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters... comprising the first sub-set of parameters and the second sub-set of parameters", recites a backward pass that uses a larger subset of network parameters than the subset of network parameters used in the forward pass.” Applicant continues on by stating that in both Evci and Xu, the size of the parameters in the forward pass is the same as that of the backward pass.
Examiner respectfully disagrees. While Evci does not explicitly state whether or not the parameter size of the forward pass has to be equal to that of the backward pass, for argument’s sake, suppose that indeed Evci does only teach the case whether the forward pass and the backward pass have equal parameter sizes. Under the broadest reasonable interpretation, “a subset of a largest of the plurality of network parameters” can mean the entire set of parameters. Furthermore, “a subset of next largest of the plurality of network parameters” can mean an empty set of parameters. This is because in this case, the total number of parameters is bipartitioned into a complete set and an empty set. Therefore, even if Examiner were to concede that Evci does indeed only teach the case where the forward and backward pass have the same size parameters, the amended limitation does not overcome the art.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6, 9-15, 22-29 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-6, 9-15, 28-29 are method claims. Claims 22-27 are machine/system/product claims. Therefore, claims 1-6, 9-15, 22-27 are directed to either a process, machine, manufacture or composition of matter.
With respect to claim 1:
Step 2A – Prong 1:
…
…
determining a forward-pass set of network parameters by selecting a first sub-set of parameters from the plurality of network parameters and setting all other parameters of the forward-pass set of network parameters to zero, wherein the first sub-set of parameters comprises a subset of a largest of the plurality of network parameters; (mental process – a person can manually select a first sub-set of parameters from the plurality of network parameters and set all other parameters of the forward-pass set of network parameters to zero with the assistance of a pen/paper.)
processing a training data item … in accordance with the forward-pass set of network parameters to generate the neural network output; (mental process – a person can manually process a training data item in accordance with the forward-pass set of network parameters to generate the neural network output with the assistance of a paper/pen.)
determining a value of an objective function from the neural network output and the training data item; (mental process – a person can manually determine a value of an objective function from the neural network output and the training data item with the assistance of a pen/paper.)
selecting a second sub-set of parameters from the plurality of network parameters, wherein the second sub-set of parameters comprises a subset of next largest of the plurality of network parameters; (mental process – a person can manually select a second sub-set of parameters from the plurality of network parameters with the assistance of a pen/paper.)
determining a backward-pass set of network parameters comprising the first sub-set of parameters and the second sub-set of parameters; (mental process – a person can manually determine a backward-pass set of network parameters comprising the first sub-set of parameters and the second sub-set of parameters with the assistance of a pen/paper.)
and updating parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters, using a gradient estimate determined from the value of the objective function. (mental process – a person can manually update parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters, using a gradient estimate determined from the value of the objective function with the assistance of a pen/paper.)
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
A computer-implemented method of … (mere instructions to apply the exception using a generic computer component – computer applies exception.)
… training a neural network having a plurality of network parameters and being configured to process an input data item to generate a neural network output, the method comprising, repeatedly: (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level recitation of training a neural network having a plurality of network parameters and being configured to process an input data item to generate a neural network output.);
…
… using the neural network … (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level recitation of processing a training data item in accordance with the forward-pass set of network parameters to generate the neural network output, using the neural network.);
With respect to claim 2:
Step 2A – Prong 1:
…
selecting the first sub-set of parameters and selecting the second sub-set of parameters; (mental process – a person can manually select the first sub-set of parameters and select the second sub-set of parameters with the assistance of a pen/paper.)
and performing, on the neural network training hardware, the repeated steps of: processing a training data item using the neural network in accordance with the forward-pass set of network parameters, and updating parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters. (mental process – a person can manually process a training data item using the neural network in accordance with the forward-pass set of network parameters, and update parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters with the assistance of a pen/paper.)
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
The method of claim 1 further comprising: apportioning the method between a general purpose processor and neural network training hardware, and performing, on the general purpose processor, the repeated steps of: (mere instructions to apply the exception using a generic computer component – general purpose processor and neural network training hardware apply exception.)
With respect to claim 3:
Step 2A – Prong 1:
The method of claim 2, wherein the plurality of network parameters is never loaded into the neural network training hardware at the same time. (mental process – a person can recognize that the plurality of network parameters is never loaded into the neural network training hardware at the same time.)
With respect to claim 4:
Step 2A – Prong 1:
The method of claim 2, wherein the neural network comprises a plurality of neural network layers, the method comprising performing the repeated steps … one neural network layer at a time. (mental process – a person can recognize that the neural network comprises a plurality of neural network layers, the method comprising performing the repeated steps on the general purpose processor one neural network layer at a time.)
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
… on the general purpose processor … (mere instructions to apply the exception using a generic computer component – general purpose processor applies exception.)
With respect to claim 5:
Step 2A – Prong 1:
The method of any of claim 2, comprising performing the repeated steps on the neural network training hardware multiple times before performing the repeated steps on the general purpose processor. (mental process – a person can recognize that the repeated steps are performed on the neural network training hardware multiple times before performing the repeated steps on the general purpose processor.)
With respect to claim 6:
Step 2A – Prong 1:
The method of claim 2, comprising, whilst repeating the method, performing the repeated steps … in parallel. (mental process – a person can recognize that the repeated steps are performed on the general purpose processor, and on the neural network training hardware, in parallel.)
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
… on the general purpose processor, and on the neural network training hardware, … (mere instructions to apply the exception using a generic computer component – general purpose processor and neural network training hardware apply exception.)
With respect to claim 9:
Step 2A – Prong 1:
The method of claim 1, wherein the neural network comprises a plurality of neural network layers, the method further comprising selecting one or both of the first sub-set of parameters and the second sub-set of parameters layer-by-layer of the neural network. (mental process – a person can recognize that the neural network comprises a plurality of neural network layers, the method further comprising selecting one or both of the first sub-set of parameters and the second sub-set of parameters layer-by-layer of the neural network.)
With respect to claim 10:
Step 2A – Prong 1:
The method of claim 1, wherein at least one of the processing of the training data item, the determining the objective function and updating parameters of the plurality of network parameters is performed on dedicated training hardware, wherein the first sub-set of parameters comprises a subset of the largest of the plurality of network parameters, wherein at least a determination of the largest of the plurality of network parameters is performed on a processor separate from the training hardware. (mental process – a person can recognize that the determining the objective function and updating parameters of the plurality of network parameters is performed on dedicated training hardware, wherein the first sub-set of parameters comprises a subset of the largest of the plurality of network parameters, wherein at least a determination of the largest of the plurality of network parameters is performed on a processor separate from the training hardware.)
With respect to claim 11:
Step 2A – Prong 1:
The method of claim 1, wherein the objective function includes a regularization term comprising one or more of: a term that penalizes parameters of the second sub-set more than parameters of the first sub-set; (mental process – a person can recognize that the objective function includes a regularization term comprising one or more of: a term that penalizes parameters of the second sub-set more than parameters of the first sub-set.)
and a term that penalizes parameters of the first sub-set more than parameters of the plurality of network parameters that are not in the first sub-set or in the second sub-set. (mental process – a person can recognize that the objective function includes a term that penalizes parameters of the first sub-set more than parameters of the plurality of network parameters that are not in the first sub-set or in the second sub-set.)
With respect to claim 12:
Step 2A – Prong 1:
The method of claim 1, wherein the objective function includes a regularization term comprising a term or terms that penalize parameters of the first sub-set and the second sub-set more than parameters that are not in either of the first sub-set or the second sub-set. (mental process – a person can recognize that the objective function includes a regularization term comprising a term or terms that penalize parameters of the first sub-set and the second sub-set more than parameters that are not in either of the first sub-set or the second sub-set.)
With respect to claim 13:
Step 2A – Prong 1:
The method of claim 1, wherein there is no updating of all the plurality of network parameters using the gradient estimate. (mental process – a person can recognize that there is no updating of all the plurality of network parameters using the gradient estimate.)
With respect to claim 14:
Step 2A – Prong 1:
The method of claim 1, wherein at least one of the first sub-set and the second sub-set is selected to meet a predetermined sparsity criterion. (mental process – a person can recognize that at least one of the first sub-set and the second sub-set is selected to meet a predetermined sparsity criterion.)
With respect to claim 15:
Step 2A – Prong 1:
The method of claim 1, further comprising, once a predetermined quality criterion has been fulfilled, stopping the repeating and, thereafter, training the neural network using as only non-zero parameters of the plurality of parameters a last selected first sub-set of parameters of the plurality of network parameters. (mental process – a person can recognize that once a predetermined quality criterion has been fulfilled, stopping the repeating and, thereafter, training the neural network using as only non-zero parameters of the plurality of parameters a last selected first sub-set of parameters of the plurality of network parameters.)
Claim 22 is substantially similar to claim 1, but has the following additional elements:
With respect to claim 22:
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
A system comprising: one or more computers; (mere instructions to apply the exception using a generic computer component – computer applies exception.)
and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for (mere instructions to apply the exception using a generic computer component – storage devices apply exception.)
Claims 23-26 are rejected on the same grounds under 35 U.S.C. 101 as claims 2-5, as they are substantially similar, respectively. Mutatis mutandis.
Claim 27 is substantially similar to claim 1, but has the following additional elements:
With respect to claim 27:
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for (mere instructions to apply the exception using a generic computer component – storage media and computers apply exception.)
With respect to claim 28:
Step 2A – Prong 1:
The method of claim 1, further comprising: … process a process a data item, … to generate a network output. (mental process – a person can manually process a data item to generate an output with the assistance of a pen/paper.)
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
… using the trained neural network to … in accordance with trained values of the plurality of parameters of the neural network, (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level recitation of using the trained neural network to process a data item to generate an output.);
With respect to claim 29:
Step 2A – Prong 1:
The method of claim 28, wherein the trained values of the plurality of parameters of the neural network are represented by a sparse data structure. (mental process – a person can recognize that the trained values of the plurality of parameters of the neural network are represented by a sparse data structure.)
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-6, 9-15, 22-24, 26-29 are rejected under 35 U.S.C. 103 as being unpatentable over Evci et al. (“Rigging the Lottery: Making All Tickets Winners”) hereinafter known as Evci in view of Xu et al. (US 20220067527 A1) hereinafter known as Xu.
Regarding independent claim 1, Evci teaches:
A computer-implemented method of training a neural network having a plurality of network parameters and being configured to process an input data item to generate a neural network output, the method comprising, repeatedly: determining a forward-pass set of network parameters by selecting a first sub-set of parameters from the plurality of network parameters and setting all other parameters of the forward-pass set of network parameters to zero, wherein the first sub-set of parameters comprises a subset of a largest of the plurality of network parameters; (Evci [Page 1, Paragraph 1]: “method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods” Evci teaches training a neural network with sparse parameter counts. Evci [Page 3, Paragraph 4]: “At regularly spaced intervals our method removes a fraction of connections based on weight magnitudes … There are many ways of distributing the non-zero weights across the layers” Evci teaches selecting a subset of parameters based on the magnitude of the weights and removing the unselected weights. Evci [Page 14, Paragraph 2]: “removing the connection θi corresponds to setting it to zero” Evci teaches that the removing of the unselected weights is equivalent to setting the values to zero. Evci [Page 18, Paragraph 4]: “forward pass: Calculating the loss of the current set of parameters on a given batch of data … We omit FLOPs needed for dropping the lowest magnitude connections in our calculations” Evci teaches that this selection and removal of weights is done during the forward pass of the neural network.)
processing a training data item using the neural network in accordance with the forward-pass set of network parameters to generate the neural network output; (Evci [Page 18, Paragraph 4]: “forward pass: Calculating the loss of the current set of parameters on a given batch of data. During this process layer activations are calculated in sequence using the previous activations and the parameters of the layer.” Evci teaches that the forward pass involves calculating the loss of the current set of parameters on a given batch of data.)
determining a value of an objective function from the neural network output and the training data item; (Evci [Page 3, Paragraph 4]: “After updating the connectivity, training continues with the updated network until the next update … train a neural network to minimize the loss function” Evci teaches that this process of updating the parameters and training is done iterative to minimize the loss function.)
selecting a second sub-set of parameters from the plurality of network parameters, wherein the second sub-set of parameters comprises a subset of next largest of the plurality of network parameters; (Evci [Page 3, Paragraph 4]: “After updating the connectivity, training continues with the updated network until the next update … train a neural network to minimize the loss function” Evci teaches that since this process is iterative and the subset is always changing because of the loss and addition of weights, there is a second subset that is selected in another iteration.)
Evci does not explicitly teach:
determining a backward-pass set of network parameters comprising the first sub-set of parameters and the second sub-set of parameters;
and updating parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters, using a gradient estimate determined from the value of the objective function.
However, Xu teaches:
determining a backward-pass set of network parameters comprising the first sub-set of parameters and the second sub-set of parameters; (Xu [¶ 0046]: “the system masking out weights in that layer that are less than the corresponding layer's threshold in each forward pass. In the corresponding backward pass (of the training iteration), the gradients of the sparsified weights may be pruned using the same mask” Xu teaches that in the backward pass, the gradients may be pruned using the same mask as that of the forward pass. Since the process is iterative, this comprises the first and second subset of parameters of the forward pass.)
and updating parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters, using a gradient estimate determined from the value of the objective function. (Xu [¶ 0055]: “Alternatively, to progress to the next training iteration, a backward pass of the training iteration may be performed, for instance, by applying a loss function 645 to the activations determined 625 for the corresponding layer and by updating 650 weights of the full-precision version of that layer based on results of the loss function” Xu teaches updating the weights of the layer during the backward pass by using the gradients from the loss function.)
Evci and Xu are in the same field of endeavor as the present invention, as the
references are directed to sparsifying neural networks by setting certain parameters to zero. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine selecting a subset of values and setting the others to zero in the forward pass as taught in Evci with selecting a subset of values and setting the others to zero in the backward pass as taught in Xu. Xu provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Evci to include teachings of Xu because the combination would allow for the neural network to be sparsified in both the forward and backward pass, enabling the adjusted neural network to be trained in any training scenario. This has the potential benefit of speeding up the training of the neural network as the insignificant parameters are removed and thus the computation can be focused only on the weights that matter.
Regarding dependent claim 2, Evci and Xu teach:
The method of claim 1
Xu teaches:
further comprising: apportioning the method between a general purpose processor and neural network training hardware, and performing, on the general purpose processor, the repeated steps of: selecting the first sub-set of parameters and selecting the second sub-set of parameters; (Xu [¶ 0123]: “prune the weights of the particular layer to remove the subset of weights from the particular layer during a forward pass of a particular one of the training iterations, where removing the subset of weights results in a sparsified subset of weights remaining in the particular layer” Xu teaches that the subset of weights from a layer of the forward pass are pruned. This is occurring during the training iterations, which show that the first subset and second subset of parameters are selected for pruning. Xu [¶ 0027]: “The preprocessing system 105 may include one or more data processing devices 210 (e.g., a central processing units (CPUs)” Xu teaches that this process is occurring by way of a CPU.)
and performing, on the neural network training hardware, the repeated steps of: … and updating parameters of the plurality of network parameters corresponding to the backward-pass set of network parameters. (Xu [¶ 0027]: “The preprocessing system 105 may include … hardware accelerators (e.g., machine learning accelerators, matrix arithmetic accelerators, etc.)” Xu teaches that the various processes are delegated to operate on the machine learning accelerating hardware.)
Evci teaches:
… processing a training data item using the neural network in accordance with the forward-pass set of network parameters, … (Evci [Page 18, Paragraph 4]: “forward pass: Calculating the loss of the current set of parameters on a given batch of data … We omit FLOPs needed for dropping the lowest magnitude connections in our calculations” Evci teaches that the forward pass involves calculating the loss of the current set of parameters on a given batch of data.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 3, Evci and Xu teach:
The method of claim 2,
Xu teaches:
wherein the plurality of network parameters is never loaded into the neural network training hardware at the same time. (Xu [¶ 0122]: “Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results” Xu teaches that the loading of the network parameters into the neural network training hardware can be done in a sequential order instead of in parallel, which means that they need not be loaded at the same time.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 5, Evci and Xu teach:
The method of any of claim 2,
Xu teaches:
comprising performing the repeated steps on the neural network training hardware multiple times before performing the repeated steps on the general purpose processor. (Xu [¶ 0027]: “The preprocessing system 105 may include one or more data processing devices 210 (e.g., a central processing units (CPUs) … hardware accelerators (e.g., machine learning accelerators, matrix arithmetic accelerators, etc.)” Xu teaches that the iterative training process can be operated by way of CPU and machine learning accelerator. Xu [¶ 0122]: “Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results” Xu teaches that the iteration of the training of the neural network can be done in a sequential order instead of in parallel, which means that they can be done one neural network layer at a time, and it may be done on the neural network hardware prior to the CPU.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 6, Evci and Xu teach:
The method of claim 2,
Xu teaches:
comprising, whilst repeating the method, performing the repeated steps on the general purpose processor, and on the neural network training hardware, in parallel. (Xu [¶ 0122]: “In certain circumstances, multitasking and parallel processing may be advantageous.” Xu teaches that the parallel processing of the neural network hardware and the CPU may be used.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 9, Evci and Xu teach:
The method of claim 1,
Xu teaches:
wherein the neural network comprises a plurality of neural network layers, the method further comprising selecting one or both of the first sub-set of parameters and the second sub-set of parameters layer-by-layer of the neural network. (Xu [¶ 0123]: “prune the weights of the particular layer to remove the subset of weights from the particular layer during a forward pass of a particular one of the training iterations, where removing the subset of weights results in a sparsified subset of weights remaining in the particular layer” Xu teaches that the subset of weights from a layer of the forward pass are pruned. This is occurring during the training iterations, which show that the first subset and second subset of parameters are selected for pruning. This is done layer by layer of the neural network.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 10, Evci and Xu teach:
The method of claim 1,
Xu teaches:
wherein at least one of the processing of the training data item, the determining the objective function and updating parameters of the plurality of network parameters is performed on dedicated training hardware, (Xu [¶ 0027]: “The preprocessing system 105 may include … hardware accelerators (e.g., machine learning accelerators, matrix arithmetic accelerators, etc.)” Xu teaches that the training is delegated to operate on the machine learning accelerating hardware.)
Evci teaches:
wherein the first sub-set of parameters comprises a subset of the largest of the plurality of network parameters, wherein at least a determination of the largest of the plurality of network parameters is performed on a processor separate from the training hardware. (Evci [Page 5, Paragraph 2]: “1. Initialize the set TK = {}. 2. Materialize a subset of size M of the full gradient … 3. Update TK to contain the Top-K elements” Evci teaches that the first sub-set of parameters is the weakly largest subset compared to the other iterations because in every iteration, the algorithm selects the top-K elements. This determination is done independent from the training as it is a pre-defined top k value in the algorithm.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 11, Evci and Xu teach:
The method of claim 1,
Evci teaches:
wherein the objective function includes a regularization term comprising one or more of: a term that penalizes parameters of the second sub-set more than parameters of the first sub-set; (Evci [Page 8, Paragraph 4]: “We train the next step prediction task with the standard cross entropy loss, the Adam optimizer, a learning rate of 7e − 4, an L2 regularization coefficient of 5e – 4” Evci teaches that the objective function includes a L2 regularization term. This penalizes the parameters of the second sub-set more than those of the first sub-set when the first has greater weights, which may be a common occurrence.)
a term that penalizes parameters of the second sub-set more than parameters of the plurality of network parameters that are not in the first sub-set or in the second sub-set; (Evci [Page 8, Paragraph 4]: “We train the next step prediction task with the standard cross entropy loss, the Adam optimizer, a learning rate of 7e − 4, an L2 regularization coefficient of 5e – 4” Evci teaches that the objective function includes a L2 regularization term. This penalizes the parameters of the second sub-set more than any other parameters when those parameters are greater in magnitude.)
and a term that penalizes parameters of the first sub-set more than parameters of the plurality of network parameters that are not in the first sub-set or in the second sub-set. (Evci [Page 8, Paragraph 4]: “We train the next step prediction task with the standard cross entropy loss, the Adam optimizer, a learning rate of 7e − 4, an L2 regularization coefficient of 5e – 4” Evci teaches that the objective function includes a L2 regularization term. This penalizes the parameters of the first sub-set more than any other parameters when those parameters are greater in magnitude.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 12, Evci and Xu teach:
The method of claim 1,
Evci teaches:
wherein the objective function includes a regularization term comprising a term or terms that penalize parameters of the first sub-set and the second sub-set more than parameters that are not in either of the first sub-set or the second sub-set. (Evci [Page 8, Paragraph 4]: “We train the next step prediction task with the standard cross entropy loss, the Adam optimizer, a learning rate of 7e − 4, an L2 regularization coefficient of 5e – 4” Evci teaches that the objective function includes a L2 regularization term. This penalizes the parameters of the first and second sub-set more than any other parameters when those parameters are greater in magnitude.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 13, Evci and Xu teach:
The method of claim 1,
Xu teaches:
wherein there is no updating of all the plurality of network parameters using the gradient estimate. (Xu [¶ 0055]: “Alternatively, to progress to the next training iteration, a backward pass of the training iteration may be performed, for instance, by applying a loss function 645 to the activations determined 625 for the corresponding layer and by updating 650 weights of the full-precision version of that layer based on results of the loss function” Xu teaches updating the weights of the layer during the backward pass by using the gradients from the loss function. Note that the weights do not necessarily have to be updated, especially if the gradient descent travels in a parallel, rather than an orthogonal direction, to the optimal point.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 14, Evci and Xu teach:
The method of claim 1,
Evci teaches:
wherein at least one of the first sub-set and the second sub-set is selected to meet a predetermined sparsity criterion. (Evci [Page 5, Paragraph 2]: “1. Initialize the set TK = {}. 2. Materialize a subset of size M of the full gradient … 3. Update TK to contain the Top-K elements” Evci teaches that the first and the second subsets are selected to meet a predetermined top k size. This shows that the extent of the sparsity is predetermined.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 15, Evci and Xu teach:
The method of claim 1,
Evci teaches:
further comprising, once a predetermined quality criterion has been fulfilled, stopping the repeating and, thereafter, training the neural network using as only non-zero parameters of the plurality of parameters a last selected first sub-set of parameters of the plurality of network parameters. (Evci [Page 5, Paragraph 2]: “1. Initialize the set TK = {}. 2. Materialize a subset of size M of the full gradient … 3. Update TK to contain the Top-K elements” Evci teaches that once the top k elements are selected and the neural network is pruned via setting the other elements to zero, the non-zero elements are used for the next iteration.)
The reasons to combine are substantially similar to those of claim 1.
Claim 22 is substantially similar to claim 1, but has the following additional elements:
Regarding independent claim 22, Evci and Xu teach:
A system comprising: one or more computers; (Xu [¶ 0027]: “The preprocessing system 105 may include one or more data processing devices 210 (e.g., a central processing units (CPUs)” Xu teaches that the system contains a computer or a CPU.)
and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for (Xu [¶ 0099]: “The storage 1058 may include instructions 1082 in the form of software, firmware, or hardware commands to implement the techniques described herein” Xu teaches that the storage contains instructions to carry out the training of the neural network.)
The reasons to combine are substantially similar to those of claim 1.
Claims 23-24, 26 are rejected on the same grounds under 35 U.S.C. 103 as claims 2-3, 5 as they are substantially similar, respectively. Mutatis mutandis.
Claim 27 is substantially similar to claim 1, but has the following additional elements:
Regarding independent claim 27, Evci and Xu teach:
One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for (Xu [¶ 0099]: “The storage 1058 may include instructions 1082 in the form of software, firmware, or hardware commands to implement the techniques described herein” Xu teaches that the storage contains instructions to carry out the training of the neural network.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 28, Evci and Xu teach:
The method of claim 1, further comprising:
Evci teaches:
using the trained neural network to process a process a data item, in accordance with trained values of the plurality of parameters of the neural network, to generate a network output. (Evci [Page 5, Figure 2]: Evci teaches that the trained neural network was used to perform a classification task on the ImageNet-2012 dataset. That is, the neural network was used to generate an output that is a classification on the assessment of an image.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 29, Evci and Xu teach:
The method of claim 28,
wherein the trained values of the plurality of parameters of the neural network are represented by a sparse data structure. (Evci [Page 3, Paragraph 2]: “idea of allowing the parameter budget to shift between different layers of the model, allowing for non-uniform sparsity. This allows the model to distribute parameters where they are most effective.” Evci teaches that the parameters shift around/between the layers of the neural network to create sparsity. The layers of the neural network at the data structures that allow for this sparsity.)
The reasons to combine are substantially similar to those of claim 1.
Claims 4, 25 are rejected under 35 U.S.C. 103 as being unpatentable over Evci in view of Xu in view of Dylski (“What are forward and backward passes in neural networks?”) hereinafter known as Dylski.
Regarding dependent claim 4, Evci and Xu teach:
The method of claim 2,
Evci and Xu do not explicitly teach:
wherein the neural network comprises a plurality of neural network layers, the method comprising performing the repeated steps on the general purpose processor one neural network layer at a time.
However, Dylski teaches:
wherein the neural network comprises a plurality of neural network layers, the method comprising performing the repeated steps on the general purpose processor one neural network layer at a time. (Dylski [Page 2, Paragraph 5]: “The forward pass refers to calculation process, values of the output layers from the inputs data. Its traversing through all neurons from first to last layer. … And then backward pass refers to process of counting changes in weights (de facto learning), using gradient descent algorithm. Computation is made from last layer, backward to the first layer.” Dylski teaches that the repeated steps, which are the forward and backward pass, are done one neural network layer at a time. More specifically, the forward pass is done neuron by neuron from the first to the last layer while the backward pass is done from last layer to the first layer.)
Dylski is in the same field of endeavor as the present invention, since it is directed to the forward and backward passes of neural network computations. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine sparsifying neural network in both the forward and backward pass in Evci as modified by Xu with doing this forward and backward operation one layer at a time as taught in Dylski. Dylski provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Evci as modified by Xu to include teachings of Dylski because the combination would allow for the operations of the forward and backward passes in the neural network to work in a sequential, layer by layer, order. This has the benefit of reassessing the sparsification of the neural network at each layer, enabling the determination of which neurons of the next layer of the forward/backward pass requires to be set to zero.
Claim 25 is rejected on the same grounds under 35 U.S.C. 103 as claim 4 as they are substantially similar. Mutatis mutandis.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU HYUNG HAN whose telephone number is (703) 756-5529. The examiner can normally be reached on MF 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kyu Hyung Han/
Examiner
Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
