DETAILED ACTION
This action is in response to claims filed 09 February 2026 for application 17806837 filed 14 June 2022. Currently claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09 February 2026 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-5, 7-10, 12-16 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20210350239) in view of Yous et al. (US 20190340511) and further in view of Min et al. (DropNet: Reducing Neural Network Complexity via Iterative Pruning).
Regarding claims 1 and 12, Wang discloses: A non-transitory machine-readable medium having executable instructions to cause one or more processing units to perform a method determine a reduced space neural network architecture, the method comprising:
receiving a full space neural network architecture, wherein the full space architecture includes a first plurality of nodes and a set of weights (“a processor, in connection with the interface and the memory, configured to submit the signals and training data into the artificial neural network including a sequence of layers, wherein each layer includes a set of neuron nodes, wherein a pair of neuron nodes from neighboring layers are mutually connected with a plural of trainable parameters to pass the signals from the previous layer to next layer,” claim 1);
transforming the set of weights … (“wherein the processor executes: the random number generator configured to modify output signals of each of the neuron nodes for regularization in a stochastic manner following a multi-dimensional distribution across layer depth and node width directions of the artificial neural network,” claim 1);
creating a second plurality of nodes less than the first plurality of nodes, wherein the second plurality of nodes is created… (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1); and
creating the reduced space neural network architecture using the second plurality of nodes (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1).
However, Wang does not explicitly disclose:
Transforming the set of weights into a transformed space;
Ranking the transformed set of weights in the transformed space.
Iteratively by:
(i) reducing the number of nodes in the first plurality of nodes to form a subnetwork of the neural network using ranking of associated weights in the transformed set of weights,
(ii) evaluating output accuracy of the subnetwork, and
(iii) further reducing the number of nodes by repeating (i) and (ii) for as long as output accuracy of the subnetwork equals or exceeds a threshold accuracy;
Yous teaches: transforming the set of weights into a transformed space (“A method, comprising: determining a transformation parameter for re-ordering a weight space of an inference model; and generating a transformed weight space based in part on re-ordering weights in the weight space and the transformation parameter.” [0105]);
ranking the transformed set of weights in the transformed space (“A method, comprising: determining a transformation parameter for re-ordering a weight space of an inference model; and generating a transformed weight space based in part on re-ordering weights in the weight space and the transformation parameter.” [0105]).
Wang and Yous are in the same field of endeavor of sparsifying and pruning neural networks and are analogous. Wang discloses a method of pruning based on transformed weights. Yous teaches transformation of weights into a transformed space and reordering the weights in the transformed space to enforce sparsity. Min teaches an iterative pruning method using ranking and a threshold accuracy. It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the known pruning method of Wang with the known transformation and ordering as disclosed by Yous to yield predictable results.
Min teaches: Iteratively by:
(i) reducing the number of nodes in the first plurality of nodes to form a subnetwork of the neural network using ranking of associated weights in the transformed set of weights,
(ii) evaluating output accuracy of the subnetwork, and
(iii) further reducing the number of nodes by repeating (i) and (ii) for as long as output accuracy of the subnetwork equals or exceeds a threshold accuracy;
(p2 algorithm 1 and figure 1, Min discloses an iterative training and pruning method wherein each iteration a subgroup of nodes is kept based on an importance derived from expected value. The loop is iterated while an accuracy is above or equal to a threshold.)
Wang, Yous and Min are in the same field of endeavor of sparsifying and pruning neural networks and are analogous. Wang discloses a method of pruning based on transformed weights. Yous teaches transformation of weights into a transformed space and reordering the weights in the transformed space to enforce sparsity. Min teaches an iterative pruning method. It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the pruning method with transformation and ranking as taught by Wang and Yous with the known iterative process as taught by Min to yield predictable results.
Regarding claims 2 and 13, Wang discloses: The non-transitory machine-readable medium of claim 1, further comprising:
regularizing the set of weights (“wherein the processor executes: the random number generator configured to modify output signals of each of the neuron nodes for regularization in a stochastic manner following a multi-dimensional distribution across layer depth and node width directions of the artificial neural network,” claim 1).
Regarding claims 3 and 14, Wang discloses: The non-transitory machine-readable medium of claim 2, wherein the transforming the set of weights further comprises: computing an activation for each of the first plurality of nodes using the regularized set of weights (“The dropout technique itself has been widely used to regularize over-parameterized deep neural networks. The role of dropout is to improve generalization performance by preventing activations from becoming strongly correlated, which in turn leads to over-training. In the standard dropout implementation, network activations are discarded (by zeroing the activation for that neuron node) during training (and testing for some embodiments) with independent probability p. A recent theory provides a viable interpretation of dropout as a Bayesian inference approximation.” [0046], “In some embodiments, the method and system of the present invention adopt some other related regularization methods; e.g., DropConnect, Drop-Block, StochasticDepth, DropPath, ShakeDrop, SpatialDrop, ZoneOut, Shake-Shake regularization, and data-driven drop in addition to regular dropout. In order to facilitate the rateless property for stochastic bottleneck AE architectures, yet another embodiment introduces an additional regularization mechanism referred to as TailDrop, as one realization of StochasticWidth.” [0047]).
Regarding claims 4 and 15, Wang discloses: The non-transitory machine-readable medium of claim 3, wherein the reducing comprises: determining a threshold, wherein the first plurality of nodes includes a first set of neurons; removing a neuron from the first set of neurons when based on a comparison of the activation and the threshold (“The dropout technique itself has been widely used to regularize over-parameterized deep neural networks. The role of dropout is to improve generalization performance by preventing activations from becoming strongly correlated, which in turn leads to over-training. In the standard dropout implementation, network activations are discarded (by zeroing the activation for that neuron node) during training (and testing for some embodiments) with independent probability p. A recent theory provides a viable interpretation of dropout as a Bayesian inference approximation.” [0046], “In some embodiments, the method and system of the present invention adopt some other related regularization methods; e.g., DropConnect, Drop-Block, StochasticDepth, DropPath, ShakeDrop, SpatialDrop, ZoneOut, Shake-Shake regularization, and data-driven drop in addition to regular dropout. In order to facilitate the rateless property for stochastic bottleneck AE architectures, yet another embodiment introduces an additional regularization mechanism referred to as TailDrop, as one realization of StochasticWidth.” [0047]).
Regarding claims 5 and 16, Wang discloses: The non-transitory machine-readable medium of claim 4, wherein the full space neural network architecture includes a plurality of layers and a neuron is removed from at least two different layers of the plurality of layers (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1).
Regarding claims 7 and 18, Wang discloses: The non-transitory machine-readable medium of claim 1, wherein the first plurality of nodes includes a first set of inputs and further comprising: reducing the first set of inputs using the … set of weights to create second set of inputs (“and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1).
Wang does not explicitly disclose ranked weights, however, Yous teaches ranked weights (“A method, comprising: determining a transformation parameter for re-ordering a weight space of an inference model; and generating a transformed weight space based in part on re-ordering weights in the weight space and the transformation parameter.” [0105]).
Regarding claims 8 and 19, Wang discloses: The non-transitory machine-readable medium of claim 7, wherein the reducing the first set of input comprises: sorting the first set of inputs (“Unlike prior arts, our neural networks employ multi-dimensional non-uniform dropout rates across the network width, channel and depth such that the neuron nodes become sorted by importance. The method with stochastic bottleneck framework enables seamless rate adaptation with high reconstruction performance, without requiring optimization of predetermined latent dimensionality at training.” [0063]).
Regarding claims 9 and 20, Wang discloses: The non-transitory machine-readable medium of claim 8, further comprising: removing the lower N inputs; evaluating a model accuracy; and reducing the first set of inputs when the model accuracy is greater than or equal to a threshold (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1).
Regarding claims 10, Wang discloses: The non-transitory machine-readable medium of claim 1, further comprising: evaluating a model accuracy with the second set of nodes (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1).
Claim(s) 6, 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view Yous and Min and further in view of Uchida et al. (US 20190294955).
Regarding claims 6 and 17, Wang does not explicitly disclose: The non-transitory machine-readable medium of claim 1, wherein the transformation is a Gram-Schmidt transformation.
Uchida teaches: wherein the transformation is a Gram-Schmidt transformation (“Also, the weight vectors X.sub.1 to X.sub.B may be set so as to be a normal orthogonal base. These can be realized by generating the weight vectors X from a normal distribution in which the average is 0 and the variance is 1, and orthogonalizing the weight vectors X using Gram-Schmidt orthogonalization or the like, for example.” [0072])
Wang, Yous, Min and Uchida are in the same field of endeavor of neural networks and are analogous. Wang discloses a system of reducing a neural network by ranking importance. Min teaches an iterative pruning method using ranking and a threshold accuracy. Uchida teaches a system the uses Gram-Schmidt orthogonalization for neural network weights. It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the transformed weights as disclosed by Wang, Yous and Min with the known Gram-Schmidt transformation as taught by Uchida to yield predictable results.
Regarding claim 11, Wang discloses: A non-transitory machine-readable medium having executable instructions to cause one or more processing units to perform a method comprising:
receiving a neural network model to predict an output from a set of inputs, the neural network model including a plurality of nodes, each node is respectively associated with one or more weights (“a processor, in connection with the interface and the memory, configured to submit the signals and training data into the artificial neural network including a sequence of layers, wherein each layer includes a set of neuron nodes, wherein a pair of neuron nodes from neighboring layers are mutually connected with a plural of trainable parameters to pass the signals from the previous layer to next layer,” claim 1), wherein weights of the plurality of nodes correspond to a weight matrix ([003-34], W is a weight matrix);
ranking the plurality of nodes according to associated one or more transformed weights (“Unlike prior arts, our neural networks employ multi-dimensional non-uniform dropout rates across the network width, channel and depth such that the neuron nodes become sorted by importance. The method with stochastic bottleneck framework enables seamless rate adaptation with high reconstruction performance, without requiring optimization of predetermined latent dimensionality at training.” [0063]);
selecting a subset of the neural network wherein, starting with the number of nodes in the plurality of nodes (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1); and
generating a reduced space neural network using the selected subnetwork, wherein the reduced space neural network predicts the output from the set of inputs within a tolerance level (“the training operator configured to update the artificial neural network parameters by using the training data such that an output of the artificial neural network provides better values in a plural of objective functions; and the adaptive truncator configured to prune outputs of the neuron nodes at each layer in a compressed size of the artificial neural network to reduce computational complexity in downstream testing phase for any new incoming data.” Claim 1).
Wang does not explicitly disclose: transforming the weights of the plurality of nodes for representing the weight matrix with orthogonal basis.
Each of the plurality of nodes associated with a respective weight of the transformed set of weights;
The subnetwork is created iteratively by:
(i) eliminating one or more nodes from the plurality of nodes to form the subnetwork using ranking of associated weights in the transformed set of weights,(ii) evaluating the output accuracy of the subnetwork, and (iii) further eliminating nodes by repeating (i) and (ii) for as long as the output accuracy of the subnetwork equals or exceeds a threshold accuracy;
Yous teaches: transforming the weights of the plurality of nodes for representing the weight matrix with orthogonal basis;
each of the plurality of nodes associated with a respective weight of the transformed set of weights;
(“A method, comprising: determining a transformation parameter for re-ordering a weight space of an inference model; and generating a transformed weight space based in part on re-ordering weights in the weight space and the transformation parameter.” [0105])
Min teaches: (i) eliminating one or more nodes from the plurality of nodes to form the subnetwork using ranking of associated weights in the transformed set of weights,(ii) evaluating the output accuracy of the subnetwork, and (iii) further eliminating nodes by repeating (i) and (ii) for as long as the output accuracy of the subnetwork equals or exceeds a threshold accuracy; (p2 algorithm 1 and figure 1, Min discloses an iterative training and pruning method wherein each iteration a subgroup of nodes is kept based on an importance derived from expected value. The loop is iterated while an accuracy is above or equal to a threshold.)
Uchida teaches: transforming the weights of the plurality of nodes for representing the weight matrix with orthogonal basis (“Also, the weight vectors X.sub.1 to X.sub.B may be set so as to be a normal orthogonal base. These can be realized by generating the weight vectors X from a normal distribution in which the average is 0 and the variance is 1, and orthogonalizing the weight vectors X using Gram-Schmidt orthogonalization or the like, for example.” [0072].)
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Please see the addition of the Min reference.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC NILSSON whose telephone number is (571)272-5246. The examiner can normally be reached M-F: 7-3.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James Trujillo can be reached at (571)-272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERIC NILSSON/Primary Examiner, Art Unit 2151