Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1,2,5,9,10,13, and 17-20 have been amended. Claims 1 – 20 are pending and examined herein.
Claims 1 – 20 are rejected under 35 U.S.C. 103.
Response to Arguments
Applicant’s arguments, see pgs. 11 – 22, filed April 14th, 2026, with respect to the rejection(s) of claim(s) 1 – 20 under 103 rejection have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Wiedeman et al. (NPL: ”Decorrelative Network Architecture for Robust Electrocardiogram”), Qi et al. (U.S. Pub. 20210304056), and Fukuda et al. (NPL: ”Direct Neuron-wise Fusion of Cognate Neural Networks”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 – 2, 4-10, 12-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shamir et al. (U.S. Pub. 2021/0158156 A1) in view of Walawalkar et al. (NPL:”Online Ensemble Model Compression using Knowledge Distillation”), Wiedeman et al. (NPL: ”Decorrelative Network Architecture for Robust Electrocardiogram”), further in view of Qi et al. (U.S. Pub. 20210304056).
Regarding Claim 1, Shamir teaches
A system, comprising: a processor that executes computer-executable components stored in a non-transitory computer-readable memory, the computer-executable components comprising: an access component that accesses a deep learning ensemble configured to perform an inferencing task; ([0006] of Shamir states “The computing system includes one or more processors and one or more non-transitory computer-readable media that collectively store: an ensemble that comprises a plurality of neural networks;…. the operations include: accessing, by the computing system, one or more training examples; processing, by the computing system, each of the one or more training examples with the ensemble to obtain an ensemble output from the ensemble;” ,which the ensemble is executed to produce inferencing output.)
Sharmir does not explicitly teach
and a distillation component that iteratively distills the deep learning ensemble into a smaller deep learning ensemble configured to perform the inferencing task,
wherein the smaller deep learning ensemble is constructed sequentially beginning from an initially empty ensemble and by adding one neural network per distillation iteration, and
wherein a current distillation iteration involves training a new neural network of the smaller deep learning ensemble via a loss function that includes a similarity term which quantifies similarity between the new neural network and one or more previously trained neural networks of the smaller deep learning ensemble and penalizes the similarity to promote non-redundancy,
wherein the similarity term is computed based on internal parameter values of the one or more previously trained neural networks or hidden feature representations generated by one or more hidden layers of the one or more previously trained neural networks;
and wherein the distillation component determines whether to perform a subsequent distillation iteration based on a performance saturation threshold by comparing a current performance metric of the smaller deep learning ensemble after addition of the new neural network to a prior performance metric of the smaller deep learning ensemble before addition of the new neural network.
However, Walawalkar teaches
and a distillation component that iteratively distills the deep learning ensemble into a smaller deep learning ensemble configured to perform the inferencing task (Pg. 3 of Walawalkar states “We present a framework which enables multiple student model training using knowledge transfer from an ensemble teacher. Each of these student models represents a version of the original model compressed to different degrees. Knowledge is distilled onto each of the compressed student through the ensemble teacher and also through intermediate representations from a pseudo teacher.” Pg. 6 of Walawalkar states “During inference stage, any of the individual student models can be selected from the ensemble depending on the computational hardware constraints. In case of lenient constraints, the entire ensemble can be used with the ensemble teacher providing inference based on the learnt ensemble knowledge… The loss in every student’s representational capacity due to compression is countered, by making each student block try and learn the intermediate feature map of its corresponding pseudo teacher block. The feature map pairs are compared using traditional Mean Squared Error loss, on which the network is trained to reduce any differences between them.” Pg. 5 of Walawalkar states “For every successive student branch the number of channels in each layer of its blocks is reduced by a certain ratio with respect to the pseudo teacher. This ratio becomes higher for every new student branch created. For example, for a four student ensemble and C being number of channels in pseudo teacher, the channels in other three students are assigned to be 0.75C, 0.5C and 0.25C. The students are compressed versions of the original model to varying degrees, which still manage to maintain the original network depth. The channels in the common base block are kept the same as original whose main purpose is to provide constant low level features to all the student branches.” Walawalkar teaches creating multiple student branches (an ensemble of students) and progressively compressing them and selecting students for deployment under hardware constraints, like the distilling into a smaller ensemble.)
Wiedeman teaches that
wherein the smaller deep learning ensemble is constructed sequentially beginning from an initially empty ensemble and by adding one neural network per distillation iteration, and (Pg. 12 of Wiedeman states “Due to the pseudo-inverse computation, which requires N > D and singular value decomposition as well as the parallel model training paradigm. Hence, this mechanism cannot scale to large problems. To solve the above problem, we make the following improvements... Finally, we remove the need for training multiple networks in parallel by training networks sequentially and holding the features of previously trained models constant. After training a model, its extracted features on all training samples are saved. While training the next model, these features are loaded with the corresponding batch samples, and then used for decorrelation. As such rather than dynamically decorrelating multiple networks at once, which requires simultaneous training of all networks, we simply use the features extracted by the previously trained networks as constant values to decorrelate against. For decorrelating against multiple models, we average the modified correlation loss against all the previously trained models. Thus, the entire decorrelation loss for model k in an ensemble:
PNG
media_image1.png
51
447
media_image1.png
Greyscale
” Wiedeman trains network sequentially and a first model is trained before any previously trained model is available for decorrelation, and after that model is trained, its extracted features are saved. Each next model is then trained using the saved features of the previously trained models for decorrelation. It would have been obvious to start the smaller student ensemble as empty, add the first trained student network, and then add each later trained student network one at a time sequentially.)
wherein a current distillation iteration involves training a new neural network of the smaller deep learning ensemble via a loss function that includes a similarity term which quantifies similarity between the new neural network and one or more previously trained neural networks of the smaller deep learning ensemble and penalizes the similarity to promote non-redundancy, (Pg. 11 – 12 of Wiedeman states that “The idea and implementation of the earlier version of our decorrelation training scheme is explained in [38]. In short, the intent of decorrelation is to reduce the correlation coefficient between the features extracted from two networks in a latent space… To reduce this term during training, the decorrelation loss is defined as:
PNG
media_image2.png
54
457
media_image2.png
Greyscale
… For decorrelating against multiple models, we average the modified correlation loss against all the previously trained models. Thus, the entire decorrelation loss for model k in an ensemble:
PNG
media_image1.png
51
447
media_image1.png
Greyscale
Figure 2 illustrates the sequential training of the decorrelated ensemble. The total loss for model k is
PNG
media_image3.png
31
514
media_image3.png
Greyscale
” The correlation loss term is a similarity term computed against every previously trained model 0 through k-1 in the ensemble and it is added directly into the total training loss used to update the new model’s parameters. Minimizing the correlation loss term during training of the new model basically penalizes similarity and promotes non redundancy relative to those previously trained models, which accomplishes what the current claim intents to do.)
wherein the similarity term is computed based on internal parameter values of the one or more previously trained neural networks or hidden feature representations generated by one or more hidden layers of the one or more previously trained neural networks; (Pg. 11 of Wiedeman states “In short, the intent of decorrelation is to reduce the correlation coefficient between the features extracted from two networks in a latent space. If Z1;Z2 2 RND are the extracted features from batch X with models f1 and f2 respectively (N is the batch size, D is the latent dimension), then the Pearson correlation coefficient is found using ordinary least squares to regress a relationship between Z1 and Z2:” Pg. 12 of Wiedeman states “First, we choose the feature layer just prior to the final logistic regression for decorrelation. In most classifiers, this point represents the highest-level features, and is typically much smaller in dimension (size 64 in our networks) than that of the other layers… Figure 2 illustrates the sequential training of the decorrelated ensemble. The total loss for model k is
PNG
media_image3.png
31
514
media_image3.png
Greyscale
where y denotes the label, zj is the feature vector extracted from x by model j” Wiedeman computes its similarity term on Zk Zi, which are feature vectors extracted by each model at the feature layer just prior to the final logistic regression. These feature layers are an internal hidden layer of the classification network selected as it represents the network’s highest-level features. )
Qi teaches that
and wherein the distillation component determines whether to perform a subsequent distillation iteration based on a performance saturation threshold by comparing a current performance metric of the smaller deep learning ensemble after addition of the new neural network to a prior performance metric of the smaller deep learning ensemble before addition of the new neural network. ([0044] of Qi states “The performance metric(s) may be compared to previously obtained performance metric(s) for a previous selected set of parameter values to determine if the performance metric(s) has/have improved.” [0045] of Qi states “This evaluation of improvement may be based on a predetermined function of the change in the performance metric(s). In the case of a single performance metric evaluation, the function may be based on the difference between the current performance metric and a previous performance metric. In the case of a plurality of performance metrics being evaluated, any desirable function of the differences in performance metrics may be utilized, e.g., a weighted evaluation of the differences between the performance metrics where a different weight may be applied to different performance metrics based on a desired implementation and a relative importance of the performance metric to the optimal performance of the ML model 140. The values generated by the function of the comparison of the performance metrics may be compared to one or more threshold values to determine if a significant enough improvement in performance has been achieved.” [0046] of Qi states “In a third stage 116 of the AutoML process 110, parameter update logic is provided for modifying the baseline set of parameter values against which the performance of other sets of parameter values are evaluated, in the case that a significant enough improvement in performance is achieved. That is, if a significant enough improvement has not been achieved, then the process may be repeated with the next selected set of parameter value settings without modifying the baseline set of parameters used for the comparison. If a significant enough improvement has been achieved then the selected set of parameter values, i.e. the selected beta*i, is used to replace the previous selected set of parameter values for configuring the ML model 140. This process may be repeated until a stopping criteria is achieved, e.g., a predetermined amount of execution time occurs, a predetermined level of performance of the ML model 140 is achieved, an amount of improvement in performance of the ML model 140 is not able to be achieved, or the like.” Qi teaches comparing a current performance metric against the immediately preceding performance metric, computing the difference between them, and comparing that different to a threshold value to decide whether a significant enough improvement was achieved. It also can iterate the process until the desired criteria is reached. Applying the known threshold saturation rule here to sequentially grown ensemble would have predictably resulted in continuing to add a network when the added network produced more than a threshold performance change and stop when it falls below the saturation threshold. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Shamir, Walawalkar, Wiedeman, and Qi. Shamir teaches distilling a deep learning ensemble into a smaller neural network for inference using a distillation loss based on ensemble outputs and training data. Walawalkar teaches training student networks from ensemble teachers using a loss that includes a supervised term based on ground truth labels and distillation term based on teacher outputs. Wiedeman teaches improving ensemble training by training the models sequentially and using features from previously trained models for decorrelation, so that later trained models are less redundant with earlier trained models. Qi teaches comparing a current performance metric with a prior performance metric and using a threshold to determine whether sufficient improvement has been achieved in an iterative machine learning process. One with ordinary skill in the art would be motivated to incorporate the teachings of Walawalkar, Wiedeman, and Qi into the Shamir to reduce redundancy among the student networks by decorrelating each newly trained network from the previously trained networks, while also providing thresholds to stop the iterative process when the added network no longer provides a sufficient performance improvement. The combination would have been predictable to apply known ensemble training and iterative threshold technique to a known knowledge distillation framework to have less redundant members of a smaller student ensemble and avoid unnecessary additional training during the iteration.
Regarding claim 2, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
wherein the access component accesses a training dataset, and wherein, during the current distillation iteration, the distillation component: ([0048] of Shamir states “Specifically, as illustrated in FIG. 1A, a set of training data 162 can include a number of training pairs. Each training pair can include a training example 12 and a ground truth label 14 associated with the training example 12. In the illustrated scheme, the training example 12, is supplied as input to the single neural network 30 and also to each component model 22 a-c of the ensemble 20.”)
initializes trainable internal parameters of the new neural network; (Pg. 7 of Walawalkar states “This loss procedure makes the pseudo teacher learn alongside the compressed students as the framework doesn’t use pretrained weights of any sort. It also helps the ensemble teacher gain richer knowledge of the dataset as it incorporates combination of every student’s learnt knowledge.” Training without pretrained weight in BRI can imply that the new network’s trainable internal parameters are initialized for training.)
selects, from the training dataset, one or more training data candidates and one or more ground-truth annotations corresponding to the one or more training data candidates; ([0048] of Shamir states “Specifically, as illustrated in FIG. 1A, a set of training data 162 can include a number of training pairs. Each training pair can include a training example 12 and a ground truth label 14 associated with the training example 12. In the illustrated scheme, the training example 12, is supplied as input to the single neural network 30 and also to each component model 22 a-c of the ensemble 20.” They pair training examples with ground truth label (i.e., annotation).)
executes the new neural network on the one or more training data candidates, thereby yielding one or more first inferencing task outputs; ([0050] of Shamir states “Similarly, the single neural network 30 can process the training example 12 to generate a network output 34 based on the training example 12. In some implementations, the network output 34 can be a single output for all “labels” (e.g., true, and distilled) or multiple outputs, where each output is matched to one or more of the available labels. Further discussion in this regard is provided with reference to FIG. 1B.”)
executes the deep learning ensemble on the one or more training data candidates, thereby yielding one or more second inferencing task outputs; ([0048] of Shamir states “Each training pair can include a training example 12 and a ground truth label 14 associated with the training example 12. In the illustrated scheme, the training example 12, is supplied as input to the single neural network 30 and also to each component model 22 a-c of the ensemble 20. In some implementations, an entirety of the features of the training example 12 are supplied to each component model 22 a-c. In other implementations, to increase diversity within the ensemble 20, different overlapping or non-overlapping subsets of the features of the training example 12 can be respectively supplied to the different component models 22 a-c of the ensemble 20.” [0050] of Shamir states “The ensemble 20 can process the training example 12 to generate an ensemble output 24 based on the training example 12. For example, the ensemble output 24 can be an aggregate (e.g., average, majority vote, etc.) of the respective outputs of the component models 22 a-c.”)
updates, via backpropagation, the trainable internal parameters of the new neural network based on the loss function, ([0055] of Shamir states “The distillation loss 42 (e.g., in combination with the supervised loss 44) can be used to train the single neural network 30. For example, the distillation loss 42 (e.g., in combination with the supervised loss 44) can be backpropagated through the single neural network 30 to update the values of the parameters (e.g., weights) of the single neural network 30.”)
wherein the loss function includes a first term that quantifies errors between the one or more first inferencing task outputs and the one or more ground-truth annotations, ([0051] of Shamir states “As is generally performed for supervised learning, a supervised loss 44 can be generated based on a comparison of the network output 34 and the ground truth label 14. Thus, the supervised loss 44 can penalize differences between the network output 34 and the ground truth label 14. Or, it can be defined as a difference based loss between values computed by the teacher and the respective ones computed by the student, including logit values in the top head.”)
wherein the loss function includes a second term that quantifies errors between the one or more first inferencing task outputs and the one or more second inferencing task outputs, ([0053] of Shamir states “According to an aspect of the present disclosure, in addition or alternatively to the supervised loss 44, a distillation loss 42 can be generated based on a comparison of the network output 34 and the ensemble output 24. The distillation loss 42 can be computed for the final predictions of the network 30 and the ensemble 20 or can be computed at an earlier stage (e.g., in the logit space). Thus, the distillation loss 42 can penalize differences between the network output 34 and the ensemble output 24.”)
and wherein the loss function includes a third term that quantifies and penalizes similarities between the new neural network and the one or more neural networks of the smaller deep learning ensemble which were trained during the one or more previous distillation iterations; (Pg. 12 of Wiedeman states “Figure 2 illustrates the sequential training of the decorrelated ensemble. The total loss for model k is
PNG
media_image3.png
31
514
media_image3.png
Greyscale
where y denotes the label, zj is the feature vector extracted from x by model j”. )
and repeats respective above acts until a training termination criterion associated with the new neural network is satisfied. (Pg. 11 of Walawalkar states “For fair comparison the ensemble and each of the baseline students are trained for the same number epochs on both datasets.”)
Regarding claim 4, the rejection of claim 2 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
the third term of the loss function is based on reciprocals of distances between: hidden feature maps produced by the new neural network; and hidden feature maps produced by the one or more neural networks of the smaller deep learning ensemble which were trained during the one or more previous distillation iterations. (Pg. 12 of Wiedeman states “To reduce this term during training, the decorrelation loss is defined as:
PNG
media_image2.png
54
457
media_image2.png
Greyscale
… For decorrelating against multiple models, we average the modified correlation loss against all the previously trained models. Thus, the entire decorrelation loss for model k in an ensemble:
PNG
media_image1.png
51
447
media_image1.png
Greyscale
Figure 2 illustrates the sequential training of the decorrelated ensemble. The total loss for model k is
PNG
media_image3.png
31
514
media_image3.png
Greyscale
” While Wiedeman does not use the term “reciprocal of distances”, a distance or reciprocal distance score is a known, equivalent way to quantify closeness between feature representations. A smaller distance corresponds to greater similarity, and a reciprocal of that distance increases as the feature maps become more similar. Using a reciprocal distance for Wiedeman’s correlation or decorrelation would have been an obvious predictable variation for penalizing redundancy. )
Regarding claim 5, the rejection of claim 2 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
in response to the training termination criterion being satisfied by the new neural network, computes a current performance metric of the smaller deep learning ensemble; compares the current performance metric to a previous performance metric of the smaller deep learning ensemble that was computed during a previous distillation iteration; commences a next distillation iteration, in response to the current performance metric differing from the previous performance metric by more than the performance saturation threshold; and determines that the smaller deep learning ensemble is complete, in response to the current performance metric differing from the previous performance metric by less than the performance saturation threshold. ([0044] of Qi states “The performance metric(s) may be compared to previously obtained performance metric(s) for a previous selected set of parameter values to determine if the performance metric(s) has/have improved.” [0045] of Qi states “This evaluation of improvement may be based on a predetermined function of the change in the performance metric(s). In the case of a single performance metric evaluation, the function may be based on the difference between the current performance metric and a previous performance metric. In the case of a plurality of performance metrics being evaluated, any desirable function of the differences in performance metrics may be utilized, e.g., a weighted evaluation of the differences between the performance metrics where a different weight may be applied to different performance metrics based on a desired implementation and a relative importance of the performance metric to the optimal performance of the ML model 140. The values generated by the function of the comparison of the performance metrics may be compared to one or more threshold values to determine if a significant enough improvement in performance has been achieved.” [0046] of Qi states “In a third stage 116 of the AutoML process 110, parameter update logic is provided for modifying the baseline set of parameter values against which the performance of other sets of parameter values are evaluated, in the case that a significant enough improvement in performance is achieved. That is, if a significant enough improvement has not been achieved, then the process may be repeated with the next selected set of parameter value settings without modifying the baseline set of parameters used for the comparison. If a significant enough improvement has been achieved then the selected set of parameter values, i.e. the selected beta*i, is used to replace the previous selected set of parameter values for configuring the ML model 140. This process may be repeated until a stopping criteria is achieved, e.g., a predetermined amount of execution time occurs, a predetermined level of performance of the ML model 140 is achieved, an amount of improvement in performance of the ML model 140 is not able to be achieved, or the like.”)
Regarding claim 6, the rejection of claim 5 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
an execution component that deploys the smaller deep learning ensemble, in response to the distillation component determining that the smaller deep learning ensemble is complete. ([0022] of Shamir states “After training, the single neural network can be deployed to generate inferences. In such fashion, the single neural model can provide a superior prediction accuracy while, during training, the ensemble can serve to influence the single neural network to be more reproducible. In another example, both accuracy and reproducibility can be improved by distilling a combination of the wide model and the ensemble into a limited resources narrow model which is then deployed.”)
Regarding claim 7, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
each neural network of the smaller deep learning ensemble exhibits a smaller footprint than each neural network of the deep learning ensemble. (Pg. 5 of Walawalkar states “The students are compressed versions of the original model to varying degrees, which still manage to maintain the original network depth. The channels in the common base block are kept the same as original whose main purpose is to provide constant low level features to all the student branches.” Each student network in the compressed ensemble has a smaller parameter footprint than the original network.)
Regarding claim 8, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
the deep learning ensemble serves as a group of network heads for a common backbone network, and wherein the smaller deep learning ensemble replaces the deep learning ensemble as the group of network heads for the common backbone network. (Pg. 5 of Walawalkar states “First, the entire architecture of a given neural network is broken down into a series of layer blocks, ideally into four blocks. The first block is designated as a common base block and the rest of the blocks are replicated in parallel to create branches as shown in Figure 1. A single student model can be viewed as a series of base block and one of the branches on top of it.”)
Claims 9 – 10, 12, 16 recite substantially similar subject matter as claims 1 – 2, 4, 8 respectively, and are rejected with the same rationale, mutatis mutandis.
Claims 13 – 15, and 18 recite substantially similar subject matter as claim 5 – 7, and 5 respectively, and are rejected with the same rationale, mutatis mutandis.
Regarding claim 17, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
A computer program product for facilitating improved distillation of deep ensembles, the computer program product comprising a non-transitory computer-readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: access a deep learning ensemble configured to perform an inferencing task and a training dataset ([0006] of Shamir states “The computing system includes one or more processors and one or more non-transitory computer-readable media that collectively store: an ensemble that comprises a plurality of neural networks;…. the operations include: accessing, by the computing system, one or more training examples; processing, by the computing system, each of the one or more training examples with the ensemble to obtain an ensemble output from the ensemble;” ,which the ensemble is executed to produce inferencing output. Pg. 1 of Walawalkar states “This paper presents a novel knowledge distillation based model compression framework consisting of a student ensemble. It enables distillation of simultaneously learnt ensemble knowledge onto each of the compressed student models.”)
and deploy the smaller deep learning ensemble. (Pg. 5 – 6 of Walawalkar states “During inference stage, any of the individual student models can be selected from the ensemble depending on the computational hardware constraints. In case of lenient constraints, the entire ensemble can be used with the ensemble teacher providing inference based on the learnt ensemble knowledge.” Using the trained smaller ensemble for inference after completion of distillation corresponds to deploying the smaller deep learning ensemble.)
The rest of claim 17 recites substantially similar subject matter as claim 1 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding claim 20, the rejection of claim 17 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi teaches
wherein, for each of the previously-trained neural networks, the loss includes a distance computed between hidden activation maps of the previously-trained neural network and hidden activation maps of the new neural network. (Pg. 12 of Wiedeman states “Finally, we remove the need for training multiple networks in parallel by training networks sequentially and holding the features of previously trained models constant. After training a model, its extracted features on all training samples are saved. While training the next model, these features are loaded with the corresponding batch samples, and then used for decorrelation. As such rather than dynamically decorrelating multiple networks at once, which requires simultaneous training of all networks, we simply use the features extracted by the previously trained networks as constant values to decorrelate against. For decorrelating against multiple models, we average the modified correlation loss against all the previously trained models. Thus, the entire decorrelation loss for model k in an ensemble:
PNG
media_image1.png
51
447
media_image1.png
Greyscale
Figure 2 illustrates the sequential training of the decorrelated ensemble. The total loss for model k is
PNG
media_image3.png
31
514
media_image3.png
Greyscale
where y denotes the label, zj is the feature vector extracted from x by model j.” Wiedeman teaches comparing hidden feature maps of the new model against hidden feature maps of each previously trained model. A distance based comparison between those same hidden feature maps would have been an obvious predictable variation of Wiedeman’s correlation comparison because both quantify redundancy between the hidden feature representations. )
Claims 3, 11, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Shamir et al. (U.S. Pub. 2021/0158156 A1) in view of Walawalkar et al. (NPL:”Online Ensemble Model Compression using Knowledge Distillation”), Wiedeman et al. (NPL: ”Decorrelative Network Architecture for Robust Electrocardiogram”), Qi et al. (U.S. Pub. 20210304056), further in view of Fukuda et al. (NPL: “Direct Neuron-wise Fusion of Cognate Neural Networks”).
Regarding claim 3, the rejection of claim 2 is incorporated herein. Furthermore, the combination of Shamir, Walawalkar, Wiedeman, and Qi does not explicitly teach
wherein the third term of the loss function is based on cosine similarities between: the trainable internal parameters of the new neural network; and trainable internal parameters of the one or more neural networks of the smaller deep learning ensemble which were trained during the one or more previous distillation iterations.
However, Fukuda teaches that
wherein the third term of the loss function is based on cosine similarities between: the trainable internal parameters of the new neural network; and trainable internal parameters of the one or more neural networks of the smaller deep learning ensemble which were trained during the one or more previous distillation iterations. (Pg. 1 of Fukuda states “A fusion weight for each neuron is determined on the basis of a cosine similarity metric when connections to the neuron are regarded as a vector. Unless neurons from two cognate networks to be fused are excessively dissimilar, the fusion weight is automatically adjusted to make it large. If the characteristics of two neurons are dissimilar, smaller fusion weight is given to avoid irrelevant neurons being generated by the fusion.” Pg. 3 of Fukuda states “A neuron-wise fusion with the cosine similarities is performed by the following equation
PNG
media_image4.png
21
171
media_image4.png
Greyscale
where WAlk and WBlk are the neuron vector of l-th layer and k-th neuron of CA and CB consisting of weights and bias parameters connecting as input to each neuron as
PNG
media_image5.png
39
238
media_image5.png
Greyscale
wAlkn, wBlkn, bAlk, and bBlk are n-th weight and bias parameters of each neuron as shown in Fig 3, respectively, lk is a fusion weight for each neuron estimated from cosine similarity Dlk as
PNG
media_image6.png
100
190
media_image6.png
Greyscale
The fusion weight lk ranges 0 < lk < a, where a is a parameter for controlling the maximum value of lk and B indicates the minimum value (cutoff parameter) considered in the cosine similarity Dlk to determine the fusion weight. Instead of utilizing input connections, output connections from the neuron can be utilized as the neuron vector as” Fukuda computes consine similarity between the trainable weight bias parameters of two different neural network. It would havae been obvious to use this known parameter vector cosine similarity metric as the specific similarity function because cosine similarty between neuron or weight vectors was already an established, predictable way to quantify how similar two trained networks’ internal parameters are. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Shamir, Walawalkar, Wiedeman, Qi, and Fukuda. Shamir teaches distilling a deep learning ensemble into a smaller neural network for inference using a distillation loss based on ensemble outputs and training data. Walawalkar teaches training student networks from ensemble teachers using a loss that includes a supervised term based on ground truth labels and distillation term based on teacher outputs. Wiedeman teaches improving ensemble training by training the models sequentially and using features from previously trained models for decorrelation, so that later trained models are less redundant with earlier trained models. Qi teaches comparing a current performance metric with a prior performance metric and using a threshold to determine whether sufficient improvement has been achieved in an iterative machine learning process. Fukuda teaches computing cosine similarity between weight and bias parameter vectors of different neural networks. One with the ordinary skill in the art would have been motivated to combine the teachings of Fukuda with the combination of Shamir, Walawalkar, Wiedeman, and Qi because the combination already trains a new student network using a term that measures similarity or redundancy relative to previously trained student networks. Fukuda merely provides a known approach to measure that similarity using the networks’ own trainable weight and bias parameters. It would have been predictable combination to quantify whether the newly trained student network is internally similar to the previously trained networks and would’ve achieved same goal of reducing redundancy among ensemble models.
Claims 11, 19 recites substantially similar subject matter as claim 3 respectively, and are rejected with the same rationale, mutatis mutandis.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights