DETAILED ACTION
Remarks
Claims 1-20 have been examined and rejected. This Office action is responsive to the amendment filed on 10/28/2025, which has been entered in the above identified application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, claim 1 recites “wherein the compiler ensembles the DNN through analyzing, identifying and removing any inter-network neuron redundancy in the Nnetworks to obtain both a savings in training time constraints and a reduction in an original memory footprint”. The relationship between these elements is unclear. It is unclear whether the limitation is intended to recite ensembling through 1) analyzing, 2) identifying, 3) removing any inter-network redundancy. It is unclear whether the limitation is intended to recite ensembling through 1) analyzing any inter-network redundancy, 2) identifying the any inter-network redundancy 3) and removing the any inter-network redundancy. It is further unclear which previous limitation “to obtain” is intended to modify (ensembles, analyzing, identifying, removing) For the purposes of examination, this limitation is interpreted as: wherein the compiler ensembles the DNN through a first step of analyzing, a second step of identifying and a third step of removing any inter-network neuron redundancy in the Nnetworks, wherein a savings in training time constraints and a reduction in an original memory footprint are obtained
Regarding claims 1 and 11, the claims recite “identify and remove dead neurons from the DNN”. It is unclear whether the limitations are intended to recite steps of 1) identify 2) remove dead neurons or 1) identify dead neurons 2) remove the dead neurons. It is further unclear whether the limitations are intended to recite “remove, from the DNN, dead neurons” or “remove dead neurons, wherein the dead neurons are from the DNN”. For the purposes of examination, this limitation is interpreted as: identify dead neurons and remove, from the DNN, the dead neurons.
Regarding claims 1 and 11, the claims recite “additional DNNs of the N network other than the main DNN and the peer DNN”. It is unclear how “the N network other than the main DNN and the peer DNN” is intended to relate the previously recited N Networks. For the purposes of examination, this limitation is interpreted as: additional DNNs of an N network other than the main DNN and the peer DNN
Regarding claim 11, claim 11 recites “a plurality of neurons ni... nx, wherein each neuron ni being a computation node in a DNN” and “each neuron of the plurality of neurons ni... nx being a computation node comprised in the N networks”. It is unclear how “each neuron ni” relates to “ach neuron of the plurality of neurons ni... nx” and whether they are the same or different computation nodes the same or different network. For the purposes of examination, these limitations are interpreted as: a plurality of neurons, wherein each neuron of the plurality of neurons is a computation node.
Regarding claim 11, claim 11 further recites “utilize the plurality of inputs i...I to train the N networks to ensemble the DNN through analyzing, identifying and removing inter-network neuron redundancy in the N networks to obtain”. The relationship between these elements is unclear. It is unclear whether “to ensemble” refers the inputs, or the N networks. It is unclear whether the analyzing, identifying and removing refer to the utilizing, the training, or the ensembling. It is further unclear whether inter-network neuron redundancy is intended to refer only the removing, or whether it is intended to also refer to the analyzing and the identifying. It is unclear which previous limitation or limitations “to obtain” is intended to refer. For the purposes of examination, this limitation is interpreted as: utilize the plurality of inputs i...I to train the N networks, ensembling the DNN, a first step of analyzing, a second step of identifying, a third step of removing inter-network neuron redundancy in the N networks, and obtaining
Regarding claims 2-10 and 12-20, claims 2-10 and 12-20 are also rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for depending on an indefinite parent claim. Claims 3 and 13 recite similar “the analyzing, identifying and removing of the inter-network neuron redundancy” limitations, which are likewise rejected and interpreted. Claims 9 and 19 recite similar “the identifying and the removing of the inter-network neuron redundancy” limitations, which are likewise rejected and interpreted.
Regarding claims 4 and 14, claims 4 and 14 recite “the first step of eliminating a redundancy to train the DNN through the dead neuron analysis (DNA) and the dead neuron elimination (DNE) to logically partition the N networks into at least the main DNN and the peer DNN”. It is unclear how this limitation is intended to relate to the previously recited first step of eliminating redundancy from the main DNN and a peer DNN through a dead neuron analysis (DNA) and a dead neuron elimination (DNE). It is further unclear how to logically partition is intended to relate to the eliminating redundance and the training. For the purposes of examination, this limitation is interpreted as: carrying out a second first step of eliminating redundancy, training the DNN through the dead neuron analysis (DNA) and the dead neuron elimination (DNE), and logically partitioning the N networks into at least the main DNN and the peer DNN
Regarding claims 5 and 15, claims 5 and 15 recite “the main DNN that is free from the dead neurons”. It is unclear how this limitation whether this limitation is intended to refer to the previously recited main DNN or a different main DNN. For the purposes of examination, this limitation is interpreted as: a second main DNN that is free from the dead neurons
Claims 5 and 15 further recite “subjected to neuron dependence analysis (NDA) to identify connected neurons that are caused to fire together according to different neuron activation functions provided to a fraction of the plurality of inputs”. It is unclear as to which previous limitation “according to different neuron activation functions” is intended to refer. It is unclear as to which previous limitation “provided to a fraction of the plurality of inputs” is intended to refer. For the purposes of examination, this limitation is interpreted as: subjected to neuron dependence analysis (NDA) to identify connected neurons that are caused to fire together, wherein different neuron activation functions are provided to a fraction of the plurality of inputs.
Claims 5 and 15 further recite “connected neurons that are caused to fire together” and “the connected neurons that fire together are linked together by an edge in a neuron dependence graph (NDG)”. The relationship between these elements is unclear. For the purposes of examination, these limitations are interpreted as: first connected neurons that are caused to fire together and second connected neurons that fire together are linked together by an edge in a neuron dependence graph (NDG)
Regarding claims 8 and 18, claims 8 and 18 recite “the output combining function as inputs”. The claims do not previously recite an output combining function as inputs. For the purposes of examination, this limitation is interpreted as: an output combining function as an input.
Regarding claims 9 and 19, claims 9 and 19 recite “a combination of Python code written to interact with a tensor flow stack and C++ code written to implement the identifying and the removing of the inter-network neuron redundancy”. It is unclear whether “combination” refers to only to the Python code, or whether it is intended to refer to both the Python code and the C++ code. It is unclear whether written refers to the combination or the Python code. It is unclear whether the Python is written to interact with both the tensor flow stack and the C++ code. For the purposes of examination, this limitation is interpreted as: Python code and C++ code written to implement the identifying and removing of the inter-network neuron redundancy, wherein the Python code is written to interact with a tensor flow stack
Regarding claims 10 and 20, claims 10 and 20 recite “the plurality of neurons ni... nx comprised in the DNN”. It is unclear whether “the plurality of neurons ni... nx comprised in the DNN” refers any of the previously recited neurons of the parent claims. For the purposes of examination, this limitation is interpreted as: a second plurality of neurons ni... nx comprised in the DNN
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 3, 11, 12, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Alhalabi et al. (Alhalabi, Besher, Mohamed Medhat Gaber, and Shadi Basurra. "EnSyth: A pruning approach to synthesis of deep learning ensembles." 2019 IEEE international conference on systems, man and cybernetics (SMC). IEEE, 2019.), hereinafter Alhalabi, in view of Han et al. (Han, Song, et al. "Learning both weights and connections for efficient neural network." Advances in neural information processing systems 28 (2015)), hereinafter Han, in further view of Bian et al. (Bian, Yijun, et al. "Sub-Architecture Ensemble Pruning in Neural Architecture Search." arXiv e-prints (2019): arXiv-1910, published 10/01/2019), hereinafter Bian.
Regarding claim 11, Alhalabi teaches the claim comprising:
A system for training a Deep Neural Network ensemble, comprising: comprising a plurality of neurons ni... nx, wherein each neuron ni being a computation node in a DNN; and at least a processor in a computer that executes program code comprising a compiler which is stored in a non-transitory computer-readable medium, wherein the compiler is enabled to configure a Deep Neural Network (DNN) into N networks, including at least a main DNN of the N networks and a peer DNN of the N networks, to perform training steps, comprising (Alhalabi 1, In this paper, we propose EnSyth an approach to synthesise deep learning ensembles from a baseline model, leading to boosting up its performance, in terms of accuracy and inference time; 3 C Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model (plurality of neurons shown); • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, Our experiment has been conducted on: OS: Ubuntu Desktop 18.04.1 LTS, CPU: Intel KVM 64bit 2,400 GHz, CASH: 16384 KB, RAM:32 GB, Python:3.6.7, TensorFlow:1.10.0, Keras:2.2.4):
receive a plurality of inputs i...I by the plurality of neurons ni... nx, wherein each neuron of the plurality of neurons ni... nx being a computation node comprised in the N networks (Alhalabi 1, EnSyth works by applying multiple sets of pruning methods by varying the corresponding hyperparameters. This, in turn, leads to a pool of diverse pruned deep learning models. Using such a pool to form deep learning ensembles, will have a powerful potential of boosting up the accuracy because of the awarded diversity from varying different values for pruning method’s hyperparameters. The number of possible ensembles that can be formed is 2 m −1, where m is the number of pruned models; 3, we first introduce the topology of the feed-forward neural network models, then we explain the pruning method [37] that has been used to generate the compressed deep learning models; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 C, Synthesising sets of divers compressed models into ensemble predictions is a critical element in our approach because ensemble learning will not only allow to create better classifiers; Let m a pruned model generated by Net-Trim and the decision of the model mi about a class wj is defined as: yi,j ∈ {0, 1} where: • i = 1, 2, · · · , N : N is the number of the classifiers. • j = 1, 2, · · · , C : C is the number of classes. if mi predicted correctly a class wj then yi,j = 1 otherwise yi,j = 0. We use plurality voting [38] as a simple ensemble learning technique to synthesise the classifiers);
and utilize the plurality of inputs i...I to train the N networks to ensemble the DNN through analyzing, identifying and removing any inter-network neuron redundancy in the N networks to obtain both a savings in training time constraints and a reduction in an original memory footprint (Alhalabi abs. Deep neural networks have achieved state-of-art performance in many domains including computer vision, natural language processing and self-driving cars. However, they are very computationally expensive and memory intensive which raises significant challenges when it comes to deploy or train them on strict latency applications or resource-limited environments; we generate a set of diverse compressed deep learning models using different hyperparameters for a pruning method, after that we utilize ensemble learning to synthesise the outputs of the compressed models to compose a new pool of classifiers; 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C, we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 D, The following is a summary statistics about the size of the 36 pruned models where only the weights and biases are saved as compressed numpy arrays [42]: 1) CIFAR-10: Max:4.2 MB, Min:1.6 MB, Avg: 2.94 MB (baseline: 4.8 MB); 2) CIFAR-5: Max: 4.2 MB, Min: 1.3 MB, Avg: 2.89 MB (baseline: 4.8 MB); 3) MNIST-FASHION: Max:7.7 MB, Min: 0.88 MB, Avg: 2.56 MB (baseline: 7.7MB); 4 I, As shown in Table. II, all of the solution space models have a smaller number of trainable parameters than the baseline model in the three data sets; 4 J, with a small number of models in an ensemble, the classifier can achieve high predictability levels. This is particularly interesting when it comes to deploying those ensembles on smartphones and Internet of Things (IoT) devices, because the ensemble size is small compared to the baseline model)
However, Alhalabi fails to expressly disclose wherein the compiler is configured to identify and remove dead neurons from the DNN. In the same field of endeavor, Han teaches:
wherein the compiler is configured to identify and remove dead neurons from the DNN (Han 1, The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs; 3, Our pruning method employs a three-step process, as illustrated in Figure 2, which begins by learning the connectivity via normal network training; 3.5, After pruning connections, neurons with zero input connections or zero output connections may be safely pruned. This pruning is furthered by removing all connections to or from a pruned neuron. The retraining phase automatically arrives at the result where dead neurons will have both zero input connections and zero output connections. This occurs due to gradient descent and regularization. A neuron that has zero input connections (or zero output connections) will have no contribution to the final loss, leading the gradient to be zero for its output connection (or input connection), respectively. Only the regularization term will push the weights to zero. Thus, the dead neurons will be automatically removed during retraining)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the compiler is configured to identify and remove dead neurons from the DNN as suggested in Han into Alhalabi. Doing so would be desirable because neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy (see Han abs.). Neural networks are prone to suffer the vanishing gradient problem [25] as the networks get deeper, which makes pruning errors harder to recover for deep networks (see Han 3.3). Neural networks have become ubiquitous. While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources. For embedded mobile applications, these resource demands become prohibitive. Our goal in pruning networks is to reduce the energy required to run such large networks so they can run in real time on mobile devices. The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs. To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy (see Han 1).
However, Alhalabi in view of Han fails to expressly disclose wherein the compiler is configured to identify a set of semantically-equivalent neurons between the main DNN of the N networks and the peer DNN of the N networks, wherein the compiler is configured to extract, from the main DNN of the N networks, the set of semantically-equivalent neurons that were identified, wherein the compiler is configured to form a common sub-network based on the set of semantically-equivalent neurons that were extracted from the main DNN of the N networks, and wherein the compiler is configured to link the common sub-network to additional DNNs of the N network other than the main DNN and the peer DNN, wherein, when training the additional DNNs, the compiler is configured to refrain from training parameters of the common sub-network. In the same field of endeavor, Bian teaches:
wherein the compiler is configured to identify a set of semantically-equivalent neurons between the main DNN of the N networks and the peer DNN of the N networks, wherein the compiler is configured to extract, from the main DNN of the N networks, the set of semantically-equivalent neurons that were identified, wherein the compiler is configured to form a common sub-network based on the set of semantically-equivalent neurons that were extracted from the main DNN of the N networks, and wherein the compiler is configured to link the common sub-network to additional DNNs of the N network other than the main DNN and the peer DNN, wherein, when training the additional DNNs, the compiler is configured to refrain from training parameters of the common sub-network (Bian p. 1, To tackle the NAS ensemble pruning problems, we seek diverse sub-ensemble architectures in a smaller size yet still with the comparable performance to the original ensemble architecture without pruning; our proposed method would lead to distinct deeper architectures than the original ensemble; p. 2, We propose a NAS ensemble pruning method to seek sub-ensemble architectures in a smaller size, benefiting from an essential characteristic, i.e., diversity in ensemble learning; AdaNet attempts to train multiple weak subarchitectures with less computation cost to comprise powerful neural architectures inspired by ensemble methods; Given an ensemble architecture f(x) = P 16k6l wk · hk(x) ∈ F searched by ensemble NAS methods such as AdaNet, and a training set S = {(x1, y1), ...,(xm, ym)} where all training instances are assumed to be drawn i.i.d. (independent and identically distributed) (multiple weak subarchitectures trained independently); p. 3, Figure 1: Layers in blue and green indicate the input and output layer, respectively; prunes the less valuable sub-architectures based on certain criteria during the searching process (lines 10–11 in Algorithm 1); p. 4, To consider accuracy and diversity simultaneously, we propose another strategy, named “Pruning by Information Entropy (PIE).”; To reveal the redundancy between two sub-architectures (wi and wj ) in the ensemble architecture, the normalized variation of information (Zadeh et al. 2017), VI(wi, wj ) = 1 − I(wi; wj ) H(wi, wj ) , (12) is used to indicate the diversity between them; 7, an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures; 7, ensemble models usually benefit from diverse individual learners; we target the ensemble learning methods in NAS and propose an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures during the searching process (as, described, and shown in Fig. 1, similar subarchitecutres are pruned, with the remaining substructures linked). Three solutions are proposed as the guiding criteria in SAEP that reflects the characteristics of the ensemble architecture (i.e., PRS, PAP, and PIE) to prune the less valuable subarchitectures. Experimental results indicate that SAEP could guide diverse sub-architectures to create sub-ensemble architectures in a smaller size yet still with comparable performance to the ensemble architecture that is not pruned. Besides, PIE might lead to distinct deeper sub-architectures if diversity is not sufficient)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the compiler is configured to identify a set of semantically-equivalent neurons between the main DNN of the N networks and the peer DNN of the N networks, wherein the compiler is configured to extract, from the main DNN of the N networks, the set of semantically-equivalent neurons that were identified, wherein the compiler is configured to form a common sub-network based on the set of semantically-equivalent neurons that were extracted from the main DNN of the N networks, and wherein the compiler is configured to link the common sub-network to additional DNNs of the N network other than the main DNN and the peer DNN, wherein, when training the additional DNNs, the compiler is configured to refrain from training parameters of the common sub-network as suggested in Bian into Alhalabi in view of Han. Doing so would be desirable because Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and the remarkable capability of reducing the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computation, which might not be affordable to researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computation, an essential characteristic of diversity in ensemble methods is missed out, causing more similar sub-architectures to be gathered and potential redundancy in the final ensemble architecture. To bridge this gap, we propose a pruning method for NAS ensembles, named as “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP).” It targets to utilize diversity and achieve sub-ensemble architectures in a smaller size with comparable performance to the unpruned ensemble architectures (see Bian abs.). SAEP might lead to distinct deeper and more effective architectures than the original one if the degree of diversity is not sufficient, which could be a bonus by pruning. Experimental results demonstrate the effectiveness of the proposed method in largely reducing the number of subarchitectures in ensemble architectures and increasing diversity while maintaining the final performance. Our proposed pruning criteria could also be generalized upon other ensemble methods, which could be interesting for future exploration (see Bian p. 2).
Regarding claim 1, claim 1 contains substantially similar limitations to those found in claim 11, the only difference being utilizing by the compiler (Alhalabi abs. we generate a set of diverse compressed deep learning models using different hyperparameters for a pruning method, after that we utilize ensemble learning to synthesise the outputs of the compressed models to compose a new pool of classifiers; 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C, we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 D, The following is a summary statistics about the size of the 36 pruned models where only the weights and biases are saved as compressed numpy arrays [42]: 1) CIFAR-10: Max:4.2 MB, Min:1.6 MB, Avg: 2.94 MB (baseline: 4.8 MB); 2) CIFAR-5: Max: 4.2 MB, Min: 1.3 MB, Avg: 2.89 MB (baseline: 4.8 MB); 3) MNIST-FASHION: Max:7.7 MB, Min: 0.88 MB, Avg: 2.56 MB (baseline: 7.7MB); see also 4 I). Consequently, claim 1 is rejected for the same reasons.
Regarding claim 2, Alhalabi in view of Han in further view of Bian teaches all the limitations of claim 1, further comprising:
wherein the plurality of inputs i…I comprises one or a combination of: a tensor flow graph describing the DNN, a training dataset, a validation dataset, an ensemble cardinality and an ensemble combining function (Alhalabi 1, EnSyth works by applying multiple sets of pruning methods by varying the corresponding hyperparameters. This, in turn, leads to a pool of diverse pruned deep learning models. Using such a pool to form deep learning ensembles, will have a powerful potential of boosting up the accuracy because of the awarded diversity from varying different values for pruning method’s hyperparameters. The number of possible ensembles that can be formed is 2 m −1, where m is the number of pruned models; 3, we first introduce the topology of the feed-forward neural network models, then we explain the pruning method [37] that has been used to generate the compressed deep learning models; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 C, Synthesising sets of divers compressed models into ensemble predictions is a critical element in our approach because ensemble learning will not only allow to create better classifiers; Let m a pruned model generated by Net-Trim and the decision of the model mi about a class wj is defined as: yi,j ∈ {0, 1} where: • i = 1, 2, · · · , N : N is the number of the classifiers. • j = 1, 2, · · · , C : C is the number of classes. if mi predicted correctly a class wj then yi,j = 1 otherwise yi,j = 0. We use plurality voting [38] as a simple ensemble learning technique to synthesise the classifiers; 4 C, Our experiment has been conducted on: OS: Ubuntu Desktop 18.04.1 LTS, CPU: Intel KVM 64bit 2,400 GHz, CASH: 16384 KB, RAM:32 GB, Python:3.6.7, TensorFlow:1.10.0, Keras:2.2.4)
Regarding claim 12, claim 12 contains substantially similar limitations to those found in claim 2. Consequently, claim 12 is rejected for the same reasons.
Regarding claim 3, Alhalabi in view of Han in further view of Bian teaches all the limitations of claim 2, further comprising:
wherein the analyzing, identifying and removing of the inter-network neuron redundancy comprises: using a portion of the plurality of inputs i…I to train at least the main DNN and the peer DNN, wherein the compiler carrying out a first step of eliminating redundancy from the main DNN and the peer DNN (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles)
However, Alhalabi fails to expressly disclose through a dead neuron analysis (DNA) and a dead neuron elimination (DNE). In the same field of endeavor, Han teaches:
through a dead neuron analysis (DNA) and a dead neuron elimination (DNE) (Han 1, The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs; 3, Our pruning method employs a three-step process, as illustrated in Figure 2, which begins by learning the connectivity via normal network training; 3.5, After pruning connections, neurons with zero input connections or zero output connections may be safely pruned. This pruning is furthered by removing all connections to or from a pruned neuron. The retraining phase automatically arrives at the result where dead neurons will have both zero input connections and zero output connections. This occurs due to gradient descent and regularization. A neuron that has zero input connections (or zero output connections) will have no contribution to the final loss, leading the gradient to be zero for its output connection (or input connection), respectively. Only the regularization term will push the weights to zero. Thus, the dead neurons will be automatically removed during retraining)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated through a dead neuron analysis (DNA) and a dead neuron elimination (DNE) as suggested in Han into Alhalabi in view of Bian. Doing so would be desirable because neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy (see Han abs.). Neural networks are prone to suffer the vanishing gradient problem [25] as the networks get deeper, which makes pruning errors harder to recover for deep networks (see Han 3.3). Neural networks have become ubiquitous. While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources. For embedded mobile applications, these resource demands become prohibitive. Our goal in pruning networks is to reduce the energy required to run such large networks so they can run in real time on mobile devices. The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs. To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy (see Han 1).
Regarding claim 13, claim 13 contains substantially similar limitations to those found in claim 3. Consequently, claim 13 is rejected for the same reasons.
Claims 4-10 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Han in further view of Bian in further view of Goldsborough (Goldsborough, Peter. "A tour of tensorflow." arXiv preprint arXiv:1610.01178 (2016)).
Regarding claim 4, Alhalabi in view of Han in further view of Bian teaches all the limitations of claim 3, further comprising:
wherein the portion of the plurality of inputs i…I comprises: tensor flow, the training dataset and the validation dataset as inputs by the compiler to carry out the first step of eliminating redundancy to train the DNN to logically partition the N networks into at least the main DNN and the peer DNN, wherein redundant neurons do not contribute to any of unique outputs Ui... UI have been eliminated in both the peer DNN and in the main DNN (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, TensorFlow; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
Han further teaches:
through the dead neuron analysis (DNA) and the dead neuron elimination (DNE), wherein dead neurons that do not contribute to any of unique outputs Ui... UI have been eliminated through the DNA and the DNE (Han 1, The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs; 3, Our pruning method employs a three-step process, as illustrated in Figure 2, which begins by learning the connectivity via normal network training; 3.5, After pruning connections, neurons with zero input connections or zero output connections may be safely pruned. This pruning is furthered by removing all connections to or from a pruned neuron. The retraining phase automatically arrives at the result where dead neurons will have both zero input connections and zero output connections. This occurs due to gradient descent and regularization. A neuron that has zero input connections (or zero output connections) will have no contribution to the final loss, leading the gradient to be zero for its output connection (or input connection), respectively. Only the regularization term will push the weights to zero. Thus, the dead neurons will be automatically removed during retraining)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated through the dead neuron analysis (DNA) and the dead neuron elimination (DNE), wherein dead neurons that do not contribute to any of unique outputs Ui... UI have been eliminated through the DNA and the DNE as suggested in Han into Alhalabi. Doing so would be desirable because neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy (see Han abs.). Neural networks are prone to suffer the vanishing gradient problem [25] as the networks get deeper, which makes pruning errors harder to recover for deep networks (see Han 3.3). Neural networks have become ubiquitous. While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources. For embedded mobile applications, these resource demands become prohibitive. Our goal in pruning networks is to reduce the energy required to run such large networks so they can run in real time on mobile devices. The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs. To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy (see Han 1).
However, in view of Han in further view of Bian fails to expressly disclose the tensor flow graph. In the same field of endeavor, Goldsborough teaches:
the tensor flow graph (Goldsborough abs. Deep learning is a branch of artificial intelligence employing deep neural network architectures that has significantly advanced the state-of-the-art in computer vision, speech recognition, natural language processing and other domains. In November 2015, Google released TensorFlow, an open source deep learning software library for defining, training and deploying machine learning models; 3, machine learning algorithms may be expressed in its dataflow graph language; TensorFlow graphs are assigned to available hardware units in a local as well as distributed environment; 3 A, In TensorFlow, machine learning algorithms are represented as computational graphs; Fig. 6: An example of how common subgraph elimination; 5, The core feature of TensorBoard is the lucid visualization of computational graphs)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated the tensor flow graph as suggested in Goldsborough into in view of Han in further view of Bian. Doing so would be desirable because the major benefit of representing an algorithm in form of a graph is not only the intuitive (visual) expression of dependencies between units of a computational model, but also the fact that the definition of a node within the graph can be kept very general (see Goldsborough 3 A). Deep learning models often employ neural networks with a highly complex and intricate structure. For example, [19] reports of deep convolutional network based on the Google Inception model with more than 36,000 individual units, while [8] states that certain long short-term memory (LSTM) architectures can span over 15,000 nodes. To maintain a clear overview of such complex networks, facilitate model debugging and allow inspection of values on various levels of detail, powerful visualization tools are required. TensorBoard, a web interface for graph visualization and manipulation built directly into TensorFlow, is an example for such a tool (see Goldsborough 5).
Regarding claim 14, claim 14 contains substantially similar limitations to those found in claim 4. Consequently, claim 14 is rejected for the same reasons.
Regarding claim 5, Alhalabi in view of Han in further view of Bian in further view of Goldsborough teaches all the limitations of claim 4, further comprising:
wherein the main DNN is subjected to neuron dependence analysis (NDA) to identify connected neurons that are caused to fire together according to different neuron activation functions provided to a fraction of the plurality of inputs i…I, wherein the connected neurons that fire together are linked together by an edge in a neuron dependence graph (NDG) (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, TensorFlow; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
Han further teaches:
wherein the main DNN that is free from the dead neurons (Han 1, The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs; 3, Our pruning method employs a three-step process, as illustrated in Figure 2, which begins by learning the connectivity via normal network training; 3.5, After pruning connections, neurons with zero input connections or zero output connections may be safely pruned. This pruning is furthered by removing all connections to or from a pruned neuron. The retraining phase automatically arrives at the result where dead neurons will have both zero input connections and zero output connections. This occurs due to gradient descent and regularization. A neuron that has zero input connections (or zero output connections) will have no contribution to the final loss, leading the gradient to be zero for its output connection (or input connection), respectively. Only the regularization term will push the weights to zero. Thus, the dead neurons will be automatically removed during retraining)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the main DNN that is free from the dead neurons as suggested in Han into Alhalabi in view of Bian in further view of Goldsborough. Doing so would be desirable because neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy (see Han abs.). Neural networks are prone to suffer the vanishing gradient problem [25] as the networks get deeper, which makes pruning errors harder to recover for deep networks (see Han 3.3). Neural networks have become ubiquitous. While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources. For embedded mobile applications, these resource demands become prohibitive. Our goal in pruning networks is to reduce the energy required to run such large networks so they can run in real time on mobile devices. The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs. To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy (see Han 1).
Regarding claim 15, claim 15 contains substantially similar limitations to those found in claim 5. Consequently, claim 15 is rejected for the same reasons.
Regarding claim 6, Alhalabi in view of Han in further view of Bian in further view of Goldsborough teaches all the limitations of claim 5, further comprising:
wherein the first step of eliminating the redundancy further comprising performing (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, TensorFlow; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
Han further teaches:
such that the main DNN is free from the dead neurons (Han 1, The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs; 3, Our pruning method employs a three-step process, as illustrated in Figure 2, which begins by learning the connectivity via normal network training; 3.5, After pruning connections, neurons with zero input connections or zero output connections may be safely pruned. This pruning is furthered by removing all connections to or from a pruned neuron. The retraining phase automatically arrives at the result where dead neurons will have both zero input connections and zero output connections. This occurs due to gradient descent and regularization. A neuron that has zero input connections (or zero output connections) will have no contribution to the final loss, leading the gradient to be zero for its output connection (or input connection), respectively. Only the regularization term will push the weights to zero. Thus, the dead neurons will be automatically removed during retraining)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated such that the main DNN is free from the dead neurons as suggested in Han into Alhalabi in view of Bian in further view of Goldsborough. Doing so would be desirable because neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy (see Han abs.). Neural networks are prone to suffer the vanishing gradient problem [25] as the networks get deeper, which makes pruning errors harder to recover for deep networks (see Han 3.3). Neural networks have become ubiquitous. While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources. For embedded mobile applications, these resource demands become prohibitive. Our goal in pruning networks is to reduce the energy required to run such large networks so they can run in real time on mobile devices. The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs. To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy (see Han 1).
Bian further teaches:
performing a step of common sub-networks extraction (CSNE), such that the sets of semantically equivalent neurons that are extracted and removed from the main DNN are to be placed in the common sub-networks, such that the main DNN is free from the common sub-networks, wherein the common sub-networks comprise connected neurons that behave in a semantically equivalent way across different networks (Bian p. 1, To tackle the NAS ensemble pruning problems, we seek diverse sub-ensemble architectures in a smaller size yet still with the comparable performance to the original ensemble architecture without pruning; our proposed method would lead to distinct deeper architectures than the original ensemble; p. 2, We propose a NAS ensemble pruning method to seek sub-ensemble architectures in a smaller size, benefiting from an essential characteristic, i.e., diversity in ensemble learning; AdaNet attempts to train multiple weak subarchitectures with less computation cost to comprise powerful neural architectures inspired by ensemble methods; Given an ensemble architecture f(x) = P 16k6l wk · hk(x) ∈ F searched by ensemble NAS methods such as AdaNet, and a training set S = {(x1, y1), ...,(xm, ym)} where all training instances are assumed to be drawn i.i.d. (independent and identically distributed); p. 3, Figure 1: Layers in blue and green indicate the input and output layer, respectively; prunes the less valuable sub-architectures based on certain criteria during the searching process (lines 10–11 in Algorithm 1); p. 4, To consider accuracy and diversity simultaneously, we propose another strategy, named “Pruning by Information Entropy (PIE).”; To reveal the redundancy between two sub-architectures (wi and wj ) in the ensemble architecture, the normalized variation of information (Zadeh et al. 2017), VI(wi, wj ) = 1 − I(wi; wj ) H(wi, wj ) , (12) is used to indicate the diversity between them; 7, an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures; 7, ensemble models usually benefit from diverse individual learners; p. 7 propose an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures during the searching process. Three solutions are proposed as the guiding criteria in SAEP that reflects the characteristics of the ensemble architecture (i.e., PRS, PAP, and PIE) to prune the less valuable subarchitectures. Experimental results indicate that SAEP could guide diverse sub-architectures to create sub-ensemble architectures in a smaller size yet still with comparable performance to the ensemble architecture that is not pruned. Besides, PIE might lead to distinct deeper sub-architectures if diversity is not sufficient)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated performing a step of common sub-networks extraction (CSNE), such that the sets of semantically equivalent neurons that are extracted and removed from the main DNN are to be placed in the common sub-networks, such that the main DNN is free from the common sub-networks, wherein the common sub-networks comprise connected neurons that behave in a semantically equivalent way across different networks as suggested in Bian into Alhalabi in view of Han in further view of Goldsborough. Doing so would be desirable because Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and the remarkable capability of reducing the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computation, which might not be affordable to researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computation, an essential characteristic of diversity in ensemble methods is missed out, causing more similar sub-architectures to be gathered and potential redundancy in the final ensemble architecture. To bridge this gap, we propose a pruning method for NAS ensembles, named as “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP).” It targets to utilize diversity and achieve sub-ensemble architectures in a smaller size with comparable performance to the unpruned ensemble architectures (see Bian abs.). SAEP might lead to distinct deeper and more effective architectures than the original one if the degree of diversity is not sufficient, which could be a bonus by pruning. Experimental results demonstrate the effectiveness of the proposed method in largely reducing the number of subarchitectures in ensemble architectures and increasing diversity while maintaining the final performance. Our proposed pruning criteria could also be generalized upon other ensemble methods, which could be interesting for future exploration (see Bian p. 2).
Regarding claim 16, claim 16 contains substantially similar limitations to those found in claim 6. Consequently, claim 16 is rejected for the same reasons.
Regarding claim 7, Alhalabi in view of Han in further view of Bian in further view of Goldsborough teaches all the limitations of claim 6, further comprising:
wherein the first step of the eliminating the redundancy is followed by carrying out a second step of DNN generation, comprising utilizing by the compiler: the main DNN, and the ensemble cardinality as inputs to generate a trained DNN having (N-2) networks (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, TensorFlow; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
Han further teaches:
networks which are free of the dead neurons (Han 1, The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs; 3, Our pruning method employs a three-step process, as illustrated in Figure 2, which begins by learning the connectivity via normal network training; 3.5, After pruning connections, neurons with zero input connections or zero output connections may be safely pruned. This pruning is furthered by removing all connections to or from a pruned neuron. The retraining phase automatically arrives at the result where dead neurons will have both zero input connections and zero output connections. This occurs due to gradient descent and regularization. A neuron that has zero input connections (or zero output connections) will have no contribution to the final loss, leading the gradient to be zero for its output connection (or input connection), respectively. Only the regularization term will push the weights to zero. Thus, the dead neurons will be automatically removed during retraining)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated networks which are free of the dead neurons as suggested in Han into Alhalabi in view of Bian in further view of Goldsborough. Doing so would be desirable because neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy (see Han abs.). Neural networks are prone to suffer the vanishing gradient problem [25] as the networks get deeper, which makes pruning errors harder to recover for deep networks (see Han 3.3). Neural networks have become ubiquitous. While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources. For embedded mobile applications, these resource demands become prohibitive. Our goal in pruning networks is to reduce the energy required to run such large networks so they can run in real time on mobile devices. The model size reduction from pruning also facilitates storage and transmission of mobile applications incorporating DNNs. To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy (see Han 1).
Bian further teaches:
the common sub-networks, networks which are free of the common sub-networks (Bian p. 1, To tackle the NAS ensemble pruning problems, we seek diverse sub-ensemble architectures in a smaller size yet still with the comparable performance to the original ensemble architecture without pruning; our proposed method would lead to distinct deeper architectures than the original ensemble; p. 2, We propose a NAS ensemble pruning method to seek sub-ensemble architectures in a smaller size, benefiting from an essential characteristic, i.e., diversity in ensemble learning; AdaNet attempts to train multiple weak subarchitectures with less computation cost to comprise powerful neural architectures inspired by ensemble methods; Given an ensemble architecture f(x) = P 16k6l wk · hk(x) ∈ F searched by ensemble NAS methods such as AdaNet, and a training set S = {(x1, y1), ...,(xm, ym)} where all training instances are assumed to be drawn i.i.d. (independent and identically distributed); p. 3, Figure 1: Layers in blue and green indicate the input and output layer, respectively; prunes the less valuable sub-architectures based on certain criteria during the searching process (lines 10–11 in Algorithm 1); p. 4, To consider accuracy and diversity simultaneously, we propose another strategy, named “Pruning by Information Entropy (PIE).”; To reveal the redundancy between two sub-architectures (wi and wj ) in the ensemble architecture, the normalized variation of information (Zadeh et al. 2017), VI(wi, wj ) = 1 − I(wi; wj ) H(wi, wj ) , (12) is used to indicate the diversity between them; 7, an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures; 7, ensemble models usually benefit from diverse individual learners; p. 7 propose an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures during the searching process. Three solutions are proposed as the guiding criteria in SAEP that reflects the characteristics of the ensemble architecture (i.e., PRS, PAP, and PIE) to prune the less valuable subarchitectures. Experimental results indicate that SAEP could guide diverse sub-architectures to create sub-ensemble architectures in a smaller size yet still with comparable performance to the ensemble architecture that is not pruned. Besides, PIE might lead to distinct deeper sub-architectures if diversity is not sufficient)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated the common sub-networks, networks which are free of the common sub-networks as suggested in Bian into Alhalabi in view of Han in further view of Goldsborough. Doing so would be desirable because Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and the remarkable capability of reducing the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computation, which might not be affordable to researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computation, an essential characteristic of diversity in ensemble methods is missed out, causing more similar sub-architectures to be gathered and potential redundancy in the final ensemble architecture. To bridge this gap, we propose a pruning method for NAS ensembles, named as “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP).” It targets to utilize diversity and achieve sub-ensemble architectures in a smaller size with comparable performance to the unpruned ensemble architectures (see Bian abs.). SAEP might lead to distinct deeper and more effective architectures than the original one if the degree of diversity is not sufficient, which could be a bonus by pruning. Experimental results demonstrate the effectiveness of the proposed method in largely reducing the number of subarchitectures in ensemble architectures and increasing diversity while maintaining the final performance. Our proposed pruning criteria could also be generalized upon other ensemble methods, which could be interesting for future exploration (see Bian p. 2).
Regarding claim 17, claim 17 contains substantially similar limitations to those found in claim 7. Consequently, claim 17 is rejected for the same reasons.
Regarding claim 8, Alhalabi in view of Han in further view of Bian in further view of Goldsborough teaches all the limitations of claim 7, further comprising:
wherein the second step of DNN generation is followed by carrying out a third step of ensembling the trained DNN, comprising: aggregating by the compiler, at least the trained DNN, the main DNN, the peer DNN, and the output combining function as inputs, to generate an ensembled DNN which is free from the inter-network neuron redundancy (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, TensorFlow; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
Bian further teaches:
the common sub-networks (Bian p. 1, To tackle the NAS ensemble pruning problems, we seek diverse sub-ensemble architectures in a smaller size yet still with the comparable performance to the original ensemble architecture without pruning; our proposed method would lead to distinct deeper architectures than the original ensemble; p. 2, We propose a NAS ensemble pruning method to seek sub-ensemble architectures in a smaller size, benefiting from an essential characteristic, i.e., diversity in ensemble learning; AdaNet attempts to train multiple weak subarchitectures with less computation cost to comprise powerful neural architectures inspired by ensemble methods; Given an ensemble architecture f(x) = P 16k6l wk · hk(x) ∈ F searched by ensemble NAS methods such as AdaNet, and a training set S = {(x1, y1), ...,(xm, ym)} where all training instances are assumed to be drawn i.i.d. (independent and identically distributed); p. 3, Figure 1: Layers in blue and green indicate the input and output layer, respectively; prunes the less valuable sub-architectures based on certain criteria during the searching process (lines 10–11 in Algorithm 1); p. 4, To consider accuracy and diversity simultaneously, we propose another strategy, named “Pruning by Information Entropy (PIE).”; To reveal the redundancy between two sub-architectures (wi and wj ) in the ensemble architecture, the normalized variation of information (Zadeh et al. 2017), VI(wi, wj ) = 1 − I(wi; wj ) H(wi, wj ) , (12) is used to indicate the diversity between them; 7, an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures; 7, ensemble models usually benefit from diverse individual learners; p. 7 propose an ensemble pruning method named “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)” to reduce the redundant sub-architectures during the searching process. Three solutions are proposed as the guiding criteria in SAEP that reflects the characteristics of the ensemble architecture (i.e., PRS, PAP, and PIE) to prune the less valuable subarchitectures. Experimental results indicate that SAEP could guide diverse sub-architectures to create sub-ensemble architectures in a smaller size yet still with comparable performance to the ensemble architecture that is not pruned. Besides, PIE might lead to distinct deeper sub-architectures if diversity is not sufficient)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated the common sub-networks as suggested in Bian into Alhalabi in view of Han in further view of Goldsborough. Doing so would be desirable because Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and the remarkable capability of reducing the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computation, which might not be affordable to researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computation, an essential characteristic of diversity in ensemble methods is missed out, causing more similar sub-architectures to be gathered and potential redundancy in the final ensemble architecture. To bridge this gap, we propose a pruning method for NAS ensembles, named as “Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP).” It targets to utilize diversity and achieve sub-ensemble architectures in a smaller size with comparable performance to the unpruned ensemble architectures (see Bian abs.). SAEP might lead to distinct deeper and more effective architectures than the original one if the degree of diversity is not sufficient, which could be a bonus by pruning. Experimental results demonstrate the effectiveness of the proposed method in largely reducing the number of subarchitectures in ensemble architectures and increasing diversity while maintaining the final performance. Our proposed pruning criteria could also be generalized upon other ensemble methods, which could be interesting for future exploration (see Bian p. 2).
Regarding claim 18, claim 18 contains substantially similar limitations to those found in claim 8. Consequently, claim 18 is rejected for the same reasons.
Regarding claim 9, Alhalabi in view of Han in further view of Bian in further view of Goldsborough teaches all the limitations of claim 7, further comprising:
comprising the compiler implementing a combination of Python code written to interact with a tensor flow stack written to implement the identifying and the removing of the inter-network neuron redundancy (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, Our experiment has been conducted on: OS: Ubuntu Desktop 18.04.1 LTS, CPU: Intel KVM 64bit 2,400 GHz, CASH: 16384 KB, RAM:32 GB, Python:3.6.7, TensorFlow:1.10.0, Keras:2.2.4; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
Goldsborough further teaches:
a combination of Python code written to interact with a tensor flow stack and C++ code written (Goldsborough abs. Deep learning is a branch of artificial intelligence employing deep neural network architectures that has significantly advanced the state-of-the-art in computer vision, speech recognition, natural language processing and other domains. In November 2015, Google released TensorFlow, an open source deep learning software library for defining, training and deploying machine learning models; 3, machine learning algorithms may be expressed in its dataflow graph language; TensorFlow graphs are assigned to available hardware units in a local as well as distributed environment; 3 A, In TensorFlow, machine learning algorithms are represented as computational graphs; 3 C, To ensure a maximum of efficiency and performance of the TensorFlow execution model, a number of optimizations are built into the library. In this subsection, we examine three such improvements: common subgraph elimination, execution scheduling and finally lossy compression; Common Subgraph Elimination: An optimization performed by many modern compilers is common subexpression elimination, whereby a compiler may possibly replace the computation of an identical value two or more times by a single instance of that computation. The result is then stored in a temporary variable and reused where it was previously recalculated. Similarly, in a TensorFlow graph, it may occur that the same operation is performed on identical inputs more than once. This can be inefficient if the computation happens to be an expensive one. Moreover, it may incur a large memory overhead given that the result of that operation must be held in; memory multiple times. Therefore, TensorFlow also employs a common subexpression, or, more aptly put, common subgraph elimination pass prior to execution. For this, the computational graph is traversed and every time two or more operations of the same type (e.g. MatMul) receiving the same input tensors are encountered, they are canonicalized to only one such subgraph. The output tensor of that single operation is then redirected to all dependent nodes. Figure 6 gives an example of common subgraph elimination; 4 A, There currently exist two programming interfaces, in C++ and Python, that permit interaction with the TensorFlow backend. The Python API boasts a very rich feature set for creation and execution of computational graphs. As of this writing, the C++ interface (which is really just the core backend implementation) provides a comparatively much more limited API, allowing only to execute graphs built with Python and serialized to Google’s Protocol Buffer20 format. While there is experimental support for also building computational graphs in C++, this functionality is currently not as extensive as in Python; Fig. 6: An example of how common subgraph elimination)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated a combination of Python code written to interact with a tensor flow stack and C++ code as suggested in Goldsborough into Alhalabi in view of Han in further view of Bian. Doing so would be desirable because the major benefit of representing an algorithm in form of a graph is not only the intuitive (visual) expression of dependencies between units of a computational model, but also the fact that the definition of a node within the graph can be kept very general (see Goldsborough 3 A). Additionally, to ensure a maximum of efficiency and performance of the TensorFlow execution model, a number of optimizations are built into the library, such as common subgraph elimination. It may occur that the same operation is performed on identical inputs more than once. This can be inefficient if the computation happens to be an expensive one. Moreover, it may incur a large memory overhead (see Goldsborough 3 C). Additionally, the TensorFlow’s API is beneficial for rapid prototyping of machine learning models (see Goldsborough 4).
Regarding claim 19, claim 19 contains substantially similar limitations to those found in claim 9. Consequently, claim 19 is rejected for the same reasons.
Regarding claim 10, Alhalabi in view of Han in further view of Bian teaches all the limitations of claim 1, further comprising:
wherein the plurality of neurons ni... nx comprised in the DNN are implemented by one of or a combination of: a plurality of Processing Units (Alhalabi 1, elimination of the impact of compression on deep learning models, by producing compressed models with better predictability measures; 2, most of those pruning techniques depend on the famous L1 and L2 regularisation [5] [6] which take a very long time to cover; 3 A, the training of a network is done using xp training example; Suppose X ∈ R N×P is a one dimensional matrix represents the training samples as X = [x1, · · · , xP ], L is a layer in the network; 3 B, Net-Trim is a post-processing pruning framework, this means it prunes a network after a training process; 3 B, Net-Trim removes the redundant connections and redirect the processing to a small group of important connections; 3 C we suggest backward elimination scheme to remove the models with reduced predictability levels and find the optimal combinations that achieve better result; 3 D, Our goal is to have accurate ensembles that have the lowest possible number of models without compromising the predictability of the ensembles; Fig. 1 illustrates EnSyth, it could be summarised as: • train a baseline model; • prune the baseline model with different values of the hyperparameters to formulate the solution’s space; • synthesis deep learning ensembles that belong to the existing space; • apply backward elimination on the composed ensembles to select the best performing ensembles; 4 C, Our experiment has been conducted on: OS: Ubuntu Desktop 18.04.1 LTS, CPU: Intel KVM 64bit 2,400 GHz, CASH: 16384 KB, RAM:32 GB, Python:3.6.7, TensorFlow:1.10.0, Keras:2.2.4; 4 E, Ideally, the elimination process should be performed on the validation set, rather than the testing set; The next stage of this work is going to utilise multi-objective optimisation methods on the validation sets to find even more accurate and smaller ensembles)
However, Alhalabi in view of Han in further view of Bian fails to expressly disclose the tensor flow graph. In the same field of endeavor, Goldsborough teaches:
one of or a combination of: a plurality of Graphical Processing Units (GPUs) and a plurality of Tensor Processing Units (TPUs) (Goldsborough abs. Deep learning is a branch of artificial intelligence employing deep neural network architectures that has significantly advanced the state-of-the-art in computer vision, speech recognition, natural language processing and other domains. In November 2015, Google released TensorFlow, an open source deep learning software library for defining, training and deploying machine learning models; 3, machine learning algorithms may be expressed in its dataflow graph language; TensorFlow graphs are assigned to available hardware units in a local as well as distributed environment; 3 A, In TensorFlow, machine learning algorithms are represented as computational graphs; Fig. 6: An example of how common subgraph elimination; 3 B, there may then be more than one device, such as, for example, five independent GPUs and/or three CPUs; Devices are the smallest, most basic entities in the TensorFlow execution model. All nodes in the graph, that is, the kernel of each operation, must eventually be mapped to an available device to be executed. In practice, a device will most often be either a CPU or a GPU. However, TensorFlow supports registration of further kinds of physical execution units by the user. For example, in May 2016, Google announced its Tensor Processing Unit (TPU), a custom built ASIC (application-specific-integrated-circuit) optimized specifically for fast tensor computations [17]. It is thus understandably easy to integrate new device classes as novel hardware emerge; 5, The core feature of TensorBoard is the lucid visualization of computational graphs)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated one of or a combination of: a plurality of Graphical Processing Units (GPUs) and a plurality of Tensor Processing Units (TPUs) as suggested in Goldsborough into Alhalabi in view of Han in further view of Bian. Doing so would be desirable because the major benefit of representing an algorithm in form of a graph is not only the intuitive (visual) expression of dependencies between units of a computational model, but also the fact that the definition of a node within the graph can be kept very general (see Goldsborough 3 A). Deep learning models often employ neural networks with a highly complex and intricate structure. For example, [19] reports of deep convolutional network based on the Google Inception model with more than 36,000 individual units, while [8] states that certain long short-term memory (LSTM) architectures can span over 15,000 nodes. To maintain a clear overview of such complex networks, facilitate model debugging and allow inspection of values on various levels of detail, powerful visualization tools are required. TensorBoard, a web interface for graph visualization and manipulation built directly into TensorFlow, is an example for such a tool (see Goldsborough 5). It is noteworthy that the rise in popularity of deep learning can be traced back to only the last few years, enabled primarily by the greater availability of large data sets, containing more training examples; the efficient use of graphical processing units (GPUs) and massively parallel commodity hardware to (see Goldsborough 1).
Regarding claim 20, claim 20 contains substantially similar limitations to those found in claim 10. Consequently, claim 10 is rejected for the same reasons.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1-7, 9, 11-17, and 19. The corrections to claims 1-7 and 11-17 are approved, and the previous objections to claims 1-7 and 11-17 are respectfully withdrawn. The corrections to the claims are approved and the rejection of claims 1-20 for being directed to directed to a judicial exception without significantly more is respectfully withdrawn.
With respect to the rejection of the claims under 35 U.S.C. 112(b), as discussed in the rejection, the metes and bounds of the claimed invention are unclear. Per the MPEP 2173, the definiteness of claim language is to ensure that the scope of the claims is clear so the public is informed of the boundaries of what constitutes infringement of the patent. During examination, a claim must be given its broadest reasonable interpretation consistent with the specification as it would be interpreted by one of ordinary skill in the art. If the language of a claim, given its broadest reasonable interpretation, is such that a person of ordinary skill in the relevant art would read it with more than one reasonable interpretation, then a rejection under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph is appropriate (see MPEP 2173). As discussed in the rejection, Examiner determined that the language of claims 1, 5, and 6 was indefinite and claims 1-6 stand rejected stand rejected under 35 U.S.C. 112(b).
Regarding independent claim 1, the Applicant alleges that Alhalabi as described in the previous Office action, does not explicitly teach wherein the compiler is configured to identify and remove dead neurons from the DNN, wherein the compiler is configured to identify a set of semantically- equivalent neurons between the main DNN of the N networks and the peer DNN of the N networks, wherein the compiler is configured to extract, from the main DNN of the N networks, the set of semantically-equivalent neurons that were identified, wherein the compiler is configured to form a common sub-network based on the set of semantically-equivalent neurons that were extracted from the main DNN of the N networks, and wherein the compiler is configured to link the common sub-network to additional DNNs of the N network other than the main DNN and the peer DNN, wherein, when training the additional DNNs, the compiler is configured to refrain from training parameters of the common sub-network, as has been amended to the claim. Examiner has therefore rejected independent claim 1 under 35 U.S.C § 103 as unpatentable over Alhalabi in view of Han in further view of Bian.
Similar arguments have been presented for claim 11 and thus, Applicant’s arguments are not persuasive for the same reasons.
Applicant states that the dependent claims recite all the limitations of the independent claims, and thus, are allowable in view of the remarks set forth regarding the independent claims. However, as discussed above, Alhalabi in view of Han in further view of Bian is considered to teach the independent claims, and consequently, the dependent claims are rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Filan, Daniel, et al. "Pruned neural networks are surprisingly modular." arXiv preprint arXiv:2003.04881 (2020), see pp. 3-9.2/
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN T REPSHER III whose telephone number is (571)272-7487. The examiner can normally be reached Monday - Friday, 8AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN T REPSHER III/ Primary Examiner, Art Unit 2143
Prosecution Insights