DETAILED ACTION
This office action is responsive to the above identified application filed 11/28/2023. The application contains claims 1-18, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Russian Federation (RU) on 5/28/2021. It is noted, however, that applicant has not filed a certified copy of the PCT/RU2021/000226 Date application as required by 37 CFR 1.55.
Information Disclosure Statement
The Information Disclosure Statement with references submitted 11/18/2024, has been considered and entered into the file.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 6-10, and 17 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites the limitation "the given layer" in the last line. There is insufficient antecedent basis for this limitation in the claim. For examination purposes examiner will consider "the given layer" as "the respective layer". The dependent claims inherit the independent claim deficiency.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 11-15 and 18 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to non-statutory subject matter. During examination, the claims must be interpreted as broadly as their terms reasonably allow. In re American Academy of Science Tech Center, 367 F.3d 1359, 1369, 70 U.S.P.Q.2d 1827, 1834 (Fed. Cir. 2004). Independent claim 11 recites a “computer readable storage medium,” which is not comprehensively defined by the specification. The broadest reasonable interpretation of a claim drawn to a computer readable storage medium covers forms of transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. Transitory propagating signals are non-statutory subject matter. In re Nuijten, 500 F.3d 1346, 1356-57, 84 U.S.P.Q.2d 1495, 1502 (Fed. Cir. 2007) (transitory embodiments are not directed to statutory subject matter). See also Subject Matter Eligibility of Computer Readable Media, 1351 Off. Gaz. Pat. Office 212 (Feb. 23, 2010). Examiner suggests adding the word “non-transitory.”
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 6-8, and 11-13 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by “meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting” Published 2019 [hereinafter D1] disclosed in IDS 11/18/2024.
With regard to Claim 1,
D1 teach a method for training a neural network model in a computer system (P. 1, 1, “we propose a minimal effort back propagation method, which we call meProp, for neural network learning”), the neural network model comprising a plurality of layers each comprising one or more neurons (P. 2, 2.1, “Forward propagation of neural network models, including feedforward neural networks, RNN, LSTM, consists of linear transformations …”), the plurality of layers comprising at least one first layer and one last layer (P. 4, 2.2.1, “If there are multiple hidden layers, the top-k sparsification needs to be applied to every hidden layer, because the sparsified gradient will again be dense from one layer to another. That is, in meProp the gradients are sparsified with a top-k operation at the output of every hidden layer”), each neurons being configured to perform forward propagation of one or more input values by applying weights to the one or more input values and generating an output value based on a function applied to the sum of the weighted input values (P. 2, 2.1, “y =Wx z =σ(y) (1) (2) where W ∈ Rn×m,x ∈ Rm,y ∈ Rn,z ∈ Rn, m is the dimension of the input vector, n is the dimension of the output vector, and σ is a non-linear function”), the one or more neurons of each layer, but the last layer, of the plurality of layers being connected with the one or more neurons of a consecutive layer, and the one or more neurons of each layer, but the first layer, of the plurality of layers being connected with the one or more neurons of a preceding layer (Fig. 1, Fig. 2, P. 3, 2.1, “The forward propagation is the same as traditional forward propagation, which computes the output vector via a matrix multiplication operation between two input tensors”), such that the output of a respective neuron of a preceding layer is used as an input value of the neurons of a consecutive layer connected with the given respective neuron (Fig. 1, Fig. 2, Fig. 3, P. 2, 2.1, “Forward propagation of neural network models, including feedforward neural networks, RNN, LSTM, consists of linear transformations and non-linear transformations. For simplicity, we take a computation unit with one linear transformation and one non-linear transformation as an example y =Wx (1) z =σ(y)”, z is input to next layer) , the method comprising:
a forward propagation step including:
inputting initial input values to the neurons of the first layer (Fig. 1, Fig. 2, P. 3, Abstract, “The forward propagation is computed as usual”, 2.1, “The forward propagation is the same as traditional forward propagation”); and
performing forward propagation from the one or more neurons of the first layer through the one or more neurons of the consecutive layers, until the one or more neurons of the last layer, to obtain output values of the one or more neurons of the last layer (P. 3, 2.1, “Figure 2 shows an illustration of the computational flow of meProp. The forward propagation is the same as traditional forward propagation, which computes the output vector via a matrix multiplication operation between two input tensors”; and
a back propagation step including:
measuring errors between the output values and expected values of the one or more neurons of the last layer (P. 2, 2.1, “As for a complete neural network frameworkwith a loss L, the original back propagation computes the gradient of the parameter matrix W as …” Eq. (7)), and
for each layer, but the last layer, until the first layer, of the plurality of layers, measuring errors between the output values and expected values of the one or more neurons of the respective layer, performing back propagation by determining weight updates values for the weight of the neurons of the respective layer, based on the measured errors between the output values and expected values of the one or more neurons of the consecutive layer (P. 2-3. 2.1, “The proposed meProp selects top-k elements of the gradient ∂L/∂y to approximate the original gradient, and passes them through the gradient computation graph according to the chain rule”), and changing the weight values of the one or more neurons of the respective layer based on the weight update values (P. 1, 1, “The idea is that we compute only a very small but critical portion of the gradient information, and update only the corresponding minimal portion of the parameters in each learning step”,
wherein performing the back propagation comprises:
determining inhibited neurons and uninhibited neurons amongst the one or more neurons of the respective layer based on measurement of errors between the output values and expected values of the one or more neurons of the respective layer (P. 2, 2.1, “The proposed meProp uses approximate gradients by keeping only top-k elements based on the magnitude values. That is, only the top-k elements with the largest absolute values are kept … Eq(5)); and
changing the weight values only of the uninhibited neurons of the respective layer (P. 2, 2.1, “As a result, only k rows or columns (depending on the lay out) of the weight matrix are modified, leading to a linear reduction (k divided by the vector dimension) in the computational cost”).
With regard to Claim 2,
D1 teach the method according to claim 1, wherein determining the inhibited and uninhibited neurons amongst the one or more neurons of the respective layer comprises:
using the measured errors between the output values and expected values to determine a contribution of each neuron of the one or more neurons of the respective layer of the plurality of layers to the output values (P. 2, 2.1, “The proposed meProp uses approximate gradients by keeping only top-k elements based on the magnitude values. That is, only the top-k elements with the largest absolute values are kept … EQ(5)”, “As for a complete neural network frame work with a loss L, the original back propagation computes the gradient of the parameter matrix W as … The proposed meProp selects top-k elements of the gradient ∂L / ∂y to approximate the original gradient”) and deciding whether a respective neuron is inhibited or uninhibited based on the contribution (P. 2, 2.1, “vector v = ⟨1,2,3,−4⟩, then top2(v) = ⟨0,0,3,−4⟩. We denote the indices of vector σ′(y)’s top-k values as {t1, t2, ..., tk}(1 ≤ k ≤ n), and the approximate gradient of the parameter matrix W and input vector x is: … (5)”, neuron is uninhibited (updated) if it contribution is in the top-k; otherwise it is inhibited (set to 0)).
With regard to Claim 3,
D1 teach the method according to claim 1, further comprising:
inputting a plurality of sets of input values to the one or more neurons of the given respective layer and performing forward propagation on the one or more neurons of the given respective layer to obtain corresponding sets of output values of the one or more neurons of the respective layer (P. 4, Fig. 3, “Figure 3. An illustration of the computational flow of meProp on a mini-batch learning setting”, P. 3, 2.1, “The forward propagation is the same as traditional forward propagation, which computes the output vector via a matrix multiplication operation between two input tensors”); and
determining weight updates values for the weight of the one or more neurons of the respective layer (P. 3-4, “The proposed meProp selects top-k elements of the gradient ∂L ∂y to approximate the original gradient … Eq(9)”) and inhibited neurons and uninhibited neurons amongst the one or more neurons of the given-respective layer, based on measurement of errors between the sets of output values and corresponding sets of expected values (P. 7, 4.8, “the top-k operation is based on the averaged values of all examples in the mini-batch. In this way, the big sparse matrix of the mini-batch will have consistent sparse patterns among examples”, P. 4, 2.2.1, “If there are multiple hidden layers, the top-k sparsification needs to be applied to every hidden layer, because the sparsified gradient will again be dense from one layer to another. That is, in meProp the gradients are sparsified with a top-k operation at the output of every hidden layer”).
With regard to Claim 6,
Claim 6 is similar in scope to claim 1. Therefore it is rejected under similar rationale. D1 further teach a computer system and processor (Abstract, The code is available at https://github.com/lancopku/meProp”, P. 3, Col. 2, 2.2, “Most of the experiments on CPU are conducted on the framework coded in C# on our own. “, “We also have an implementation based on the PyTorch framework for GPU based experiments”, P. 6, 4.1, “The experiments on CPU are conducted on a computer with the INTEL(R) Xeon(R) 3.0GHz CPU. The experiments on GPU are conducted on NVIDIA GeForce GTX 1080”).
With regard to Claim 7,
Claim 7 is similar in scope to claim 2. Therefore it is rejected under similar rationale.
With regard to Claim 8,
Claim 8 is similar in scope to claim 3. Therefore it is rejected under similar rationale.
With regard to Claim 11,
Claim 11 is similar in scope to claim 1. Therefore it is rejected under similar rationale. D1 further teach a computer-readable storage medium comprising instructions (Abstract, The code is available at https://github.com/lancopku/meProp”, P. 3, Col. 2, 2.2, “Most of the experiments on CPU are conducted on the framework coded in C# on our own. “, “We also have an implementation based on the PyTorch framework for GPU based experiments”, P. 6, 4.1, “The experiments on CPU are conducted on a computer with the INTEL(R) Xeon(R) 3.0GHz CPU. The experiments on GPU are conducted on NVIDIA GeForce GTX 1080”).
With regard to Claim 12,
Claim 12 is similar in scope to claim 2. Therefore it is rejected under similar rationale.
With regard to Claim 13,
Claim 13 is similar in scope to claim 3. Therefore it is rejected under similar rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4, 9, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over “meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting” Published 2019 [hereinafter D1] disclosed in IDS 11/18/2024 in view of “SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization” Published 2017 [hereinafter D2].
With regard to Claim 4,
D1 teach the method according to claim 1, the neural network model comprising n layers, with n ≥ 2 (P. 4, 2.2.1, “If there are multiple hidden layers, the top-k sparsification needs to be applied to every hidden layer, because the sparsified gradient will again be dense from one layer to another. That is, in meProp the gradients are sparsified with a top-k operation at the output of every hidden layer”).
D1 does not teach the n layers are divided in a first group of layers gathering the first p layers, with 0<p<n , and a second group of layers gathering the last n-p layers, the method being performed separately on the first group of layers and on the second group of layers.
D2 teach neural network model comprising n layers, with n ≥ 2 (P. 7, “a 16 layer residual network”), wherein the n layers are divided in a first group of layers gathering the first p layers, with 0<p<n (P. 1, Abstract, “We propose a novel deep neural network that is both lightweight and effectively structured for model parallelization. Our network, which we name as SplitNet, automatically learns to split the network weights into either a set or a hierarchy of multiple groups that use disjoint sets of features, by learning both the class-to-group and feature-to-group assignment matrices along with the network weights. This produces a tree structured network that involves no connection between branched subtrees of semantically disparate class groups. SplitNet thus greatly re duces the number of parameters and required computations, and is also embarrassingly model parallelizable at test time, since the evaluation for each subnetwork”), and a second group of layers gathering the last n-p layers (P.5, “Algorithm 1 Splitting Deep Neural Networks”, 3.3.1, “Therefore we do not split all layers but split layers down to S-th layer (S ≤ L), while maintaining lower layers (l < S) to be shared across class groups”), the method being performed separately on the first group of layers and on the second group of layers (Abstract, “the evaluation for each subnetwork is completely independent”, P. 5, 3.3.1, “no signal is passed across different groups of layers, so that forward and backward propagation in each group is independent from the processes in other groups” ), .
D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of training neural networks specifically sparsity in deep learning. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify D1 as described above to obtains networks with significantly reduced number of parameters while achieving comparable or superior accuracies over original full deep networks, and accelerated test speed with multiple GPUs (D2, Abstract). This simply combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 9,
Claim 9 is similar in scope to claim 4. Therefore it is rejected under similar rationale.
With regard to Claim 14,
Claim 14 is similar in scope to claim 4. Therefore it is rejected under similar rationale.
Claims 5, 10, 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over “meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting” Published 2019 [hereinafter D1] disclosed in IDS 11/18/2024 in view of “SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization” Published 2017 [hereinafter D2] in view of “Lateral Inhibition-Inspired Convolutional Neural Network for Visual Attention and Saliency Detection” Published 2018 [hereinafter D3] disclosed in IDS 11/18/2024.
With regard to Claim 5,
D1-D2 teach the method according to claim 4.The same motivation to combine for claim 4 equally applies for current claim
D1-D2 does not explicitly teach the neural network model is a model for image classification in the computer system , the sets of input values corresponding to image pixels values , a connexion path from one neuron of the one or more neurons of the first layer to one neuron of the one or more neurons of the last layer in the network corresponding to a channel from one input pixel value to one output pixel value wherein determining the inhibited neurons and the uninhibited neurons amongst the one or more neurons of the respective layer, further comprises: determining a two dimensional mask-based on measurement of errors between the output values and expected values, wherein each element of the two dimensional mask corresponds to a neuron of the one or more neurons in the respective layer and comprising an-inhibition information, and wherein propagating the weight updates values to change the weight values only of the uninhibited neurons of the respective layer comprises applying the mask to avoid the weights of the inhibited neurons
D3 teach neural network model is a model for image classification in the computer system (Abstract, “we propose to formulate lateral inhibition inspired by the related studies from neurobiology, and embed it into the top-down gradient computation of a general CNN for classification”, the sets of input values corresponding to image pixels values (Figure 1, “Figure 1: LICNN for visual attention. (a) Input images. Here, three very challenging “dalmatian-on-snow” images are particularly shown”), a connexion path from one neuron of the one or more neurons of the first layer to one neuron of the one or more neurons of the last layer in the network corresponding to a channel from one input pixel value to one output pixel value (P. 2-3, “A pre-trained CNN is employed to process the input image with normal feed forward”, Our Method, Top-down Feedback Signals “specific neurons will be activated if there are positively correlated patterns lying in their receptive field… Next, we construct lateral connections between different points in the Max-C Map to compute inhibition strength for each location”, Max-C Map is a 2D mask of dimension (W, H)),
wherein determining the inhibited neurons and the uninhibited neurons amongst the one or more neurons of the respective layer, further comprises:
determining a two dimensional mask-based on measurement of errors between the output values and expected values (P. 3, “Firstly, a simple inhibition along the channel axis is per formed by selecting the maximum CW at each location, wherein each element of the two dimensional mask corresponds to a neuron of the one or more neurons in the respective layer and comprising an-inhibition information (P. 3, “Assume that the layer l produces a cub of CWs with the dimension of (W,H,C),where W, H, C denote width, height and channels, respectively. Normally the C neurons at the same location represent different patterns though they share information in the same receptive field.”), and
wherein propagating the weight updates values to change the weight values only of the uninhibited neurons of the respective layer comprises applying the mask to avoid the weights of the inhibited neurons (Eq. (3)) and allow the weights of the uninhibited neurons to be changed depending on the corresponding inhibition information (P. 4, Col. 1, “The average term protects the neurons within a high response zone”, Col. 2, “Consequently, the edges of the objects are enhanced”).
D1-D2 and D3 are analogous art to the claimed invention because they are from a similar field of endeavor of sparsity in deep learning. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1-D2 resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify D1-D2 as described above to provide discriminative category-specific attention maps derived with much less noise (D3, P.2, Col. 1). This simply combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 16,
D1-D2-D3 teach the method according to claim 5 wherein the two dimensional mask comprises one or more zero values and one or more non-zero values (D3, Eq. (3)), and the inhibited neurons are associated with the one or more zero values in the two dimensional mask at the respective layer of the plurality of layers (D3, P. 5, Algorithm 1, “4: Block the neurons across all channels on the inhibited locations by fixing them to zeros, then perform feed-forward and category-specific gradient back-propagation again;”, and the uninhibited neurons are associated with the one or more non-zero values in the two dimensional mask at the respective layer of the plurality of layers (D3, P. 4, Eq. (3), “where xij denotes a point in the Max-C Map at the location (i, j) and xLI ij is its inhibition value”). The same motivation to combine for claim 5 equally applies for current claim
With regard to Claim 10,
Claim 10 is similar in scope to claim 5. Therefore it is rejected under similar rationale.
With regard to Claim 15,
Claim 15 is similar in scope to claim 5. Therefore it is rejected under similar rationale.
With regard to Claim 17,
Claim 17 is similar in scope to claim 16. Therefore it is rejected under similar rationale.
With regard to Claim 18,
Claim 18 is similar in scope to claim 16. Therefore it is rejected under similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
“Minimal Effort Back Propagation for Convolutional Neural Networks” that disclose that as traditional neural network consumes a significant amount of computing resources during back propagation. The paper propose an improvement to previous simple yet effective technique to alleviate this problem. In this technique, only a small subset of the full gradients are computed to update the model parameters. The paper extend this technique into the Convolutional Neural Network (CNN) to reduce calculation in back propagation See at least Abstract.
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights