Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/15/2025 has been considered by the examiner.
Status of Claims
Applicant’s Amendments files on 03/16/2026 has been entered and made of record.
Currently pending Claim(s):
Amended claim(s):
Canceled claim(s):
New Claim(s):
1-18, and 21 and 22
1, 8, and 15
19 and 2021 and 22
Response to Arguments
This office actions in responsive to Applicant’s Arguments/Remarks made in an Amendment received on March 16, 2026.
Applicant’s Reply (03/16/2026) includes substantive amendments to the claims. This Office action has been updated with new grounds of rejection addressing those amendments. Further Applicant’s Arguments/Remarks with respect to independent claims 1, 8, and 15 have been considered, and Examiner would like to point out to sections where Wang teaches “align the [augmented] student knowledge data with the teacher knowledge data”; Method [page 6, section 3.1]“In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit z t . To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalize”. Further, the applicant argues in summary:
“instructions to identify a loss associated with mutual distillation and ensemble distillation of the augmented student knowledge data and the teacher knowledge data, the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits” (Applicant Argument/ Remarks Page 7-8)
The Examiner would like to point out to sections where Wang teaches “identify a loss associated with at least one of mutual distillation and ensemble distillation… loss to characterize image recognition accuracy of the neural network.” Related Work [Page 4, paragraph 1] “applied mutual learning distillation instead of the teacher-student method and achieve better performance. This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML)”; Page 14 last paragraph “the gap between the test and training error rate on CIFAR10 was much lower than on CIFAR100; then, KD methods’ generalization effect was not significant. In this case, the smoothed target of self-KD may have become more effective than KD loss in MSD…” and Page 2, last paragraph “we ensemble the backup teachers by a simple Fully Connected Network (FCN). As Figure 1 shows, by ensembling the backup output ligits, the FCN acts as the new teacher for the current student network training. whereas the FCN is only trained by the KL loss. The ensembling output is usually more accurate than the backup outputs.”, as these limitations were previously presented and taught by Wang. Further Applicant’s Argument Remarks with respect to independent claims 1, 8, and 15 but are moot because the arguments do not apply to any of the references being used in the current rejection and the arguments are now rejected by newly cited art ‘Kang et al. "Ensemble learning of lightweight deep learning models using knowledge distillation for image classification." Mathematics 8.10 (2020): 1652. (hereinafter, “Kang”)’ as explained in the body of rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 7, 8, 10-12, and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. ("Memory-replay knowledge distillation." Sensors 21.8 (2021): 2792.) (hereinafter, Wang) in view of Kang et al. "Ensemble learning of lightweight deep learning models using knowledge distillation for image classification." Mathematics 8.10 (2020): 1652. (hereinafter, “Kang”).
Regarding claim 1, Wang discloses an apparatus for knowledge distillation in a neural network, the apparatus comprising (Abstract “Knowledge Distillation (KD), which transfers the knowledge from a teacher to a student network by penalizing their Kullback–Leibler (KL) divergence, is a widely used tool for Deep Neural Network (DNN) compression in intelligent sensor systems”):
Figure 1:
PNG
media_image1.png
624
766
media_image1.png
Greyscale
at least one memory; instructions in the apparatus; and processor circuitry to execute the instructions to (Introduction [page 1, paragraph 1]” a carefully designed supernet space and model searching strategy, Neural Architecture Search(NAS) techniques [8,9] can find proper models to fit various hardware and sensor requirements (flops, memory).”):
identify a source data batch and an augmented data batch (Introduction [page 1, paragraph 2] ”Knowledge Distillation (KD) [10] is also a popular technique that has been investigated quite intensively for model compression recently. KD compressed the knowledge from the teacher model, which is a larger model or a set of multiple models, to a single small student model. The teacher model is trained stand-alone beforehand. In the procedure of student training, the teacher model’s parameters are frozen, and the Kullback–Leibler (KL) divergence loss between their output probabilities is penalized”), augmented data of the augmented data batch generated based on at least one data augmentation technique applied to the source data batch (Abstract “The role of the teacher in self-KD is usually played by multi-branch peers or the identical sample with different augmentation”);
share one or more parameters between a student neural network corresponding to the source data batch and a teacher neural network corresponding to the augmented data batch (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit z t . To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”), the one or more parameters including one or more convolution layers to be shared between the teacher neural network and the student neural network to generate student knowledge data and teacher knowledge data (Page 4, paragraph 2 “This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML) [16] method of four peers with different low-level weight sharing. We evaluate four-model DML directly and found comparable results”);
[apply the at least one data augmentation technique to generate augmented student knowledge data to] align the [augmented] student knowledge data with the teacher knowledge data (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit zt. To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”); and
identify a loss associated with mutual distillation (Related Work [Page 4, paragraph 1] “applied mutual learning distillation instead of the teacher-student method and achieve better performance. This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML)”; Page 14 last paragraph “the gap between the test and training error rate on CIFAR10 was much lower than on CIFAR100; then, KD methods’ generalization effect was not significant. In this case, the smoothed target of self-KD may have become more effective than KD loss in MSD…” and ensemble distillation [of the augmented student knowledge data and the teacher knowledge data, [the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits,] the loss to characterize image recognition accuracy of the neural network (Page 2, last paragraph “we ensemble the backup teachers by a simple Fully Connected Network (FCN). As Figure 1 shows, by ensembling the backup output ligits, the FCN acts as the new teacher for the current student network training. whereas the FCN is only trained by the KL loss. The ensembling output is usually more accurate than the backup outputs.”; (Method [page 7, section 3.2] “The proposed method can extend to n memory copies ˆθ 1 , . . ., ˆθ n , with κ training steps interval. The KL loss in Equation (5) is substituted as:
PNG
media_image2.png
60
480
media_image2.png
Greyscale
).
However, Wang fails to teach apply the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] and a loss associated with…the augmented student knowledge data and the teacher knowledge data, the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits .
Kang teaches apply the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] (Page 8 Subsection 3.1 first paragraph “overfitting. In the image augmentation step, input images are mapped into an extended space, in which all their variances are covered…Our image augmentation module consists of (1) normalizing each image by mean and standard deviation, (2) randomly cropping the image to 32 × 32 pixels size with a padding of 4, and (3) applying a random horizontal flip) and a loss associated with…the augmented student knowledge data and the teacher knowledge data (Page 8 section 3.2 first paragraph “function. The loss function consists of the student loss between the output of the student model and ground-truth label and the distilled loss between student and teacher models for each distillation”), the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data (Page 8 Subsection 3.2.1 Student Loss “the student loss can be calculated as the cross-entropy between the soft target of the student model zs estimated by softmax function and the ground-truth label as follows:
PNG
media_image3.png
120
506
media_image3.png
Greyscale
where y is a ground-truth one-hot vector which represents the ground-truth label of the training dataset as 1 and all other elements are 0, and zsi is the logit (the output of the last layer) for the i-th class of the student model.”) and (2) a loss identified based on ensemble logits (Page 9, Subsection 3.2.2 Distilled Loss of the Response-Based Model “The distilled loss of response-based model using logits [26] can be calculated using the mean square error between the logits of the student model zs and the logits of the teacher model zt as follows:
PNG
media_image4.png
118
386
media_image4.png
Greyscale
).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang’s reference to include apply the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] and a loss associated with…the augmented student knowledge data and the teacher knowledge data, the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits taught by Kang’s reference. The motivation for doing so would have been to reduce the overfitting problem on training models by augmenting the data and to combine the prediction of several individual models in order to improve robustness and increase the accuracy of the model as suggested by Kang (see Kang Page 8 Section 3.1 first paragraph and page 10 Subsection 3.3 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Kang with Wang to obtain the invention specified in claim 1.
Regarding claim 3, which claim 1 is incorporated, Wang discloses wherein the at least one data augmentation technique includes at least one of a MixUp data augmentation technique, a CutMix data augmentation technique, or an AutoAug data augmentation technique (Page 17, paragraph 1 “We also report the result of CP-ResNet combined with Mixup [45], which is a widely used augmentation method in ASC tasks.”).
Regarding claim 4, which claim 1 is incorporated, Wang discloses wherein the processor circuitry is to identify loss associated with the at least one of the mutual distillation or the ensemble distillation based on Kullback-Leibler divergence (Introduction [page 1, paragraph 2] “In the procedure of student training, the teacher model’s parameters are frozen, and the Kullback–Leibler (KL) divergence loss between their output probabilities is penalized. KD is applied to various tasks.”; Page 2, last paragraph “we ensemble the backup teachers by a simple Fully Connected Network (FCN). As Figure 1 shows, by ensembling the backup output ligits, the FCN acts as the new teacher for the current student network training… whereas the FCN is only trained by the KL loss. The ensembling output is usually more accurate than the backup outputs.”; (Method [page 7, section 3.2] “The proposed method can extend to n memory copies ˆθ 1 , . . ., ˆθ n , with κ training steps interval. The KL loss in Equation (5) is substituted as:
PNG
media_image2.png
60
480
media_image2.png
Greyscale
).
Regarding claim 5, which claim 1 is incorporated, Wang discloses wherein the processor circuitry is to train model parameters corresponding to at least one of the teacher neural network or the student neural network based on forward propagation or backward propagation (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training.).
Regarding claim 7, which claim 1 is incorporated, Wang discloses a second loss associated with the mutual distillation (Related Work [Page 4, paragraph 1] “applied mutual learning distillation instead of the teacher-student method and achieve better performance. This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML)”; Page 14 last paragraph “the gap between the test and training error rate on CIFAR10 was much lower than on CIFAR100; then, KD methods’ generalization effect was not significant. In this case, the smoothed target of self-KD may have become more effective than KD loss in MSD…”) and a third loss associated with the ensemble distillation (Page 2, last paragraph “we ensemble the backup teachers by a simple Fully Connected Network (FCN). As Figure 1 shows, by ensembling the backup output ligits, the FCN acts as the new teacher for the current student network training… whereas the FCN is only trained by the KL loss. The ensembling output is usually more accurate than the backup outputs.).
However, Wang fails to teach wherein the loss is a first loss, and the processor circuitry is to determine the first loss based on a combination [of second loss and third loss].
Kang teaches wherein the loss is a first loss, and the processor circuitry is to determine the first loss based on a combination [of second loss and third loss] (Page 9 Subsection 3.2.4 last paragraph “The total loss of relation-based model using RKD is then calculated as the joint of the distilled and student losses as follows:
PNG
media_image5.png
70
520
media_image5.png
Greyscale
)
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Wang’s reference to include the loss is a first loss, and the processor circuitry is to determine the first loss based on a combination [of second loss and third loss] taught by Kang’s reference. The motivation for doing so would have been to train the student model with the total loss to mimic a teacher model as suggested by Kang (see Kang Page 9 Subsection 3.2.4 last paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Kang with Wang to obtain the invention specified in claim 7.
Regarding claim 8, Wang discloses a method for knowledge distillation in a neural network, comprising (Abstract “Knowledge Distillation (KD), which transfers the knowledge from a teacher to a student network by penalizing their Kullback–Leibler (KL) divergence, is a widely used tool for Deep Neural Network (DNN) compression in intelligent sensor systems”):
Figure 1:
PNG
media_image1.png
624
766
media_image1.png
Greyscale
identifying a source data batch and an augmented data batch (Introduction [page 1, paragraph 2] ”Knowledge Distillation (KD) [10] is also a popular technique that has been investigated quite intensively for model compression recently. KD compressed the knowledge from the teacher model, which is a larger model or a set of multiple models, to a single small student model. The teacher model is trained stand-alone beforehand. In the procedure of student training, the teacher model’s parameters are frozen, and the Kullback–Leibler (KL) divergence loss between their output probabilities is penalized”), augmented data of the augmented data batch generated based on at least one data augmentation technique applied to the source data batch (Abstract “The role of the teacher in self-KD is usually played by multi-branch peers or the identical sample with different augmentation”);
sharing one or more parameters between a student neural network corresponding to the source data batch and a teacher neural network corresponding to the augmented data batch (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit z t . To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”), the one or more parameters including one or more convolution layers to be shared between the teacher neural network and the student neural network to generate student knowledge data and teacher knowledge data (Page 4, paragraph 2 “This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML) [16] method of four peers with different low-level weight sharing. We evaluate four-model DML directly and found comparable results”);
[apply the at least one data augmentation technique to generate augmented student knowledge data to] align the [augmented] student knowledge data with the teacher knowledge data (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit zt. To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”) and
identifying a loss associated with mutual distillation (Related Work [Page 4, paragraph 1] “applied mutual learning distillation instead of the teacher-student method and achieve better performance. This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML)”; Page 14 last paragraph “the gap between the test and training error rate on CIFAR10 was much lower than on CIFAR100; then, KD methods’ generalization effect was not significant. In this case, the smoothed target of self-KD may have become more effective than KD loss in MSD…” and ensemble distillation [of the augmented student knowledge data and the teacher knowledge data, [the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits,] the loss to characterize image recognition accuracy of the neural network (Page 2, last paragraph “we ensemble the backup teachers by a simple Fully Connected Network (FCN). As Figure 1 shows, by ensembling the backup output ligits, the FCN acts as the new teacher for the current student network training... whereas the FCN is only trained by the KL loss. The ensembling output is usually more accurate than the backup outputs.”; (Method [page 7, section 3.2] “The proposed method can extend to n memory copies ˆθ 1 , . . ., ˆθ n , with κ training steps interval. The KL loss in Equation (5) is substituted as:
PNG
media_image2.png
60
480
media_image2.png
Greyscale
).
However, Wang fails to teach applying the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] and a loss associated with…the augmented student knowledge data and the teacher knowledge data, the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits.
Kang teaches applying the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] (Page 8 Subsection 3.1 first paragraph “overfitting. In the image augmentation step, input images are mapped into an extended space, in which all their variances are covered…Our image augmentation module consists of (1) normalizing each image by mean and standard deviation, (2) randomly cropping the image to 32 × 32 pixels size with a padding of 4, and (3) applying a random horizontal flip) and a loss associated with…the augmented student knowledge data and the teacher knowledge data (Page 8 section 3.2 first paragraph “function. The loss function consists of the student loss between the output of the student model and ground-truth label and the distilled loss between student and teacher models for each distillation”), the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data (Page 8 Subsection 3.2.1 Student Loss “the student loss can be calculated as the cross-entropy between the soft target of the student model zs estimated by softmax function and the ground-truth label as follows:
PNG
media_image3.png
120
506
media_image3.png
Greyscale
where y is a ground-truth one-hot vector which represents the ground-truth label of the training dataset as 1 and all other elements are 0, and zsi is the logit (the output of the last layer) for the i-th class of the student model.”) and (2) a loss identified based on ensemble logits (Page 9, Subsection 3.2.2 Distilled Loss of the Response-Based Model “The distilled loss of response-based model using logits [26] can be calculated using the mean square error between the logits of the student model zs and the logits of the teacher model zt as follows:
PNG
media_image4.png
118
386
media_image4.png
Greyscale
).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang’s reference to include applying the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] and a loss associated with…the augmented student knowledge data and the teacher knowledge data the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits taught by Kang’s reference. The motivation for doing reduce the overfitting problem on training models by augmenting the data and to combine the prediction of several individual models in order to improve robustness and increase the accuracy of the model as suggested by Kang (see Kang Page 8 Section 3.1 first paragraph and page 10 Subsection 3.3 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Kang with Wang to obtain the invention specified in claim 8.
Regarding claim 10 (drawn to a method), claim 10 is rejected the same as claim 3 and the arguments similar to that presented above for claim 3 are equally applicable to the claim 10, and all the other limitations similar to claim 3 are not repeated herein, but incorporated by reference.
Regarding claim 11 (drawn to a method), claim 11 is rejected the same as claim 4 and the arguments similar to that presented above for claim 4 are equally applicable to the claim 11, and all the other limitations similar to claim 4 are not repeated herein, but incorporated by reference.
Regarding claim 12 (drawn to a method), claim 12 is rejected the same as claim 5 and the arguments similar to that presented above for claim 5 are equally applicable to the claim 12, and all the other limitations similar to claim 5 are not repeated herein, but incorporated by reference.
Regarding claim 14 (drawn to a method), claim 14 is rejected the same as claim 7 and the arguments similar to that presented above for claim 7 are equally applicable to the claim 14, and all the other limitations similar to claim 7 are not repeated herein, but incorporated by reference.
Regarding claim 15, Wang discloses at least one non-transitory computer readable storage medium comprising computer readable instructions which, when executed, cause one or more processors to at least (Introduction [page 1, paragraph 1]” a carefully designed supernet space and model searching strategy, Neural Architecture Search(NAS) techniques [8,9] can find proper models to fit various hardware and sensor requirements (flops, memory).”):
identify a source data batch and an augmented data batch (Introduction [page 1, paragraph 2] ”Knowledge Distillation (KD) [10] is also a popular technique that has been investigated quite intensively for model compression recently. KD compressed the knowledge from the teacher model, which is a larger model or a set of multiple models, to a single small student model. The teacher model is trained stand-alone beforehand. In the procedure of student training, the teacher model’s parameters are frozen, and the Kullback–Leibler (KL) divergence loss between their output probabilities is penalized”), augmented data of the augmented data batch generated based on at least one data augmentation technique applied to the source data batch (Abstract “The role of the teacher in self-KD is usually played by multi-branch peers or the identical sample with different augmentation”);
share one or more parameters between a student neural network corresponding to the source data batch and a teacher neural network corresponding to the augmented data batch (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit z t . To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”), the one or more parameters including one or more convolution layers to be shared between the teacher neural network and the student neural network to generate student knowledge data and teacher knowledge data (Page 4, paragraph 2 “This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML) [16] method of four peers with different low-level weight sharing. We evaluate four-model DML directly and found comparable results”);
[apply the at least one data augmentation technique to generate augmented student knowledge data to] align the [augmented] student knowledge data with the teacher knowledge data (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit zt. To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”); and
identify a loss associated with at least one of mutual distillation (Related Work [Page 4, paragraph 1] “applied mutual learning distillation instead of the teacher-student method and achieve better performance. This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML)”; Page 14 last paragraph “the gap between the test and training error rate on CIFAR10 was much lower than on CIFAR100; then, KD methods’ generalization effect was not significant. In this case, the smoothed target of self-KD may have become more effective than KD loss in MSD…” and ensemble distillation [of the augmented student knowledge data and the teacher knowledge data, [the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits,] the loss to characterize image recognition accuracy of the neural network (Page 2, last paragraph “we ensemble the backup teachers by a simple Fully Connected Network (FCN). As Figure 1 shows, by ensembling the backup output ligits, the FCN acts as the new teacher for the current student network training… whereas the FCN is only trained by the KL loss. The ensembling output is usually more accurate than the backup outputs.”; (Method [page 7, section 3.2] “The proposed method can extend to n memory copies ˆθ 1 , . . ., ˆθ n , with κ training steps interval. The KL loss in Equation (5) is substituted as:
PNG
media_image2.png
60
480
media_image2.png
Greyscale
).
However, Wang fails to teach apply the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] and a loss associated with…the augmented student knowledge data and the teacher knowledge data, the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits.
Kang teaches to teach apply the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] (Page 8 Subsection 3.1 first paragraph “overfitting. In the image augmentation step, input images are mapped into an extended space, in which all their variances are covered…Our image augmentation module consists of (1) normalizing each image by mean and standard deviation, (2) randomly cropping the image to 32 × 32 pixels size with a padding of 4, and (3) applying a random horizontal flip) and a loss associated with…the augmented student knowledge data and the teacher knowledge data (Page 8 section 3.2 first paragraph “function. The loss function consists of the student loss between the output of the student model and ground-truth label and the distilled loss between student and teacher models for each distillation”), the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data (Page 8 Subsection 3.2.1 Student Loss “the student loss can be calculated as the cross-entropy between the soft target of the student model zs estimated by softmax function and the ground-truth label as follows:
PNG
media_image3.png
120
506
media_image3.png
Greyscale
where y is a ground-truth one-hot vector which represents the ground-truth label of the training dataset as 1 and all other elements are 0, and zsi is the logit (the output of the last layer) for the i-th class of the student model.”) and (2) a loss identified based on ensemble logits (Page 9, Subsection 3.2.2 Distilled Loss of the Response-Based Model “The distilled loss of response-based model using logits [26] can be calculated using the mean square error between the logits of the student model zs and the logits of the teacher model zt as follows:
PNG
media_image4.png
118
386
media_image4.png
Greyscale
).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang’s reference to include apply the at least one data augmentation technique to generate augmented student knowledge data to [align the augmented student knowledge data with the teacher knowledge data] and a loss associated with…the augmented student knowledge data and the teacher knowledge data, the loss including (1) a loss identified between logits of the augmented student knowledge data and ground truth data and (2) a loss identified based on ensemble logits taught by Kang’s reference. The motivation for doing so would have been to reduce the overfitting problem on training models by augmenting the data and to combine the prediction of several individual models in order to improve robustness and increase the accuracy of the model as suggested by Kang (see Kang Page 8 Section 3.1 first paragraph and page 10 Subsection 3.3 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Kang with Wang to obtain the invention specified in claim 15.
Regarding claim 16 (non-transitory storage medium), claim 16 is rejected the same as claim 4 and the arguments similar to that presented above for claim 4 are equally applicable to the claim 16, and all the other limitations similar to claim 4 are not repeated herein, but incorporated by reference.
Regarding claim 17 (non-transitory storage medium), claim 17 is rejected the same as claim 5 and the arguments similar to that presented above for claim 5 are equally applicable to the claim 17, and all the other limitations similar to claim 5 are not repeated herein, but incorporated by reference.
Claims 2 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. ("Memory-replay knowledge distillation." Sensors 21.8 (2021): 2792.) (hereinafter, Wang) in view of Kang et al. "Ensemble learning of lightweight deep learning models using knowledge distillation for image classification." Mathematics 8.10 (2020): 1652. (hereinafter, “Kang”), and further in view of Wu et al. ("$ L1 $-norm batch normalization for efficient training of deep neural networks." IEEE transactions on neural networks and learning systems 30.7 (2018): 2043-2051.) (hereinafter, Wu).
Regarding claim 2, which claim 1 is incorporated, Wang discloses one or more parameters are shared between the student neural network and the teacher neural network (Method [page 6, section 3.1] “In the KD method, a teacher network is trained beforehand. The parameter of the pre-trained teacher is then frozen, and only forward-propagation is conducted during the student training. The teacher outputs a corresponding logit z t . To transfer the knowledge from the teacher model to the student, KL Divergence between their output probabilities is penalized”).
However, Wang, and Kang fail to teach wherein batch normalization layers of the teacher neural network and the student neural network are to remain separate.
Wu teaches wherein batch normalization layers of the teacher neural network and the student neural network are to remain separate (Introduction [page 2043 right column paragraph 2] “batch normalization (BN) [7] has been proposed to facilitate training by explicitly normalizing inputs of each layer to have zero mean and unit variance.”; Experiments [page 4 right column paragraph 2] “The learning rate η is set to 0.1 and divided by 10 at epoch 30 and epoch 60. As for SVHN [34] dataset, we use a VGG-like network with totally 7 layers”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang in view of Kang to include wherein batch normalization layers of the teacher neural network and the student neural network are to remain separate taught by Wu. The motivation for doing so would have been to reduce the difficulties of annealing learning rate and initializing parameters as suggested by Wu (see Wu Introduction [page 1 right column paragraph 1]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Kang and Wu with Wang to obtain the invention specified in claim 2.
Regarding claim 9 (drawn to a method), claim 9 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 9, and all the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Claims 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. ("Memory-replay knowledge distillation." Sensors 21.8 (2021): 2792.) (hereinafter, Wang) in view of Kang et al. "Ensemble learning of lightweight deep learning models using knowledge distillation for image classification." Mathematics 8.10 (2020): 1652. (hereinafter, “Kang”), and further in view of Kim et al. ("Self-knowledge distillation with progressive refinement of targets." Proceedings of the IEEE/CVF international conference on computer vision. 2021.) (hereinafter, Kim).
Regarding claim 6, which claim 1 is incorporated, Wang and Kang fail to teach wherein the at least one data augmentation technique includes a random permutation function, the random permutation function to adjust an image based on a beta distribution.
Kim teaches wherein the at least one data augmentation technique includes a random permutation function (Page 6571 [right column paragraph 1] “PS-KD is combined with Cutout or CutMix, each data selects the regularization method with a probability of 0.5. Simply, PS-KD is applied to the half of the data in a randomly shuffled mini-batch, and Cutout or CutMix is performed on another half of the data.”,
the random permutation function to adjust an image based on a beta distribution (Page 6571 [right column paragraph 1] “For existing regularization methods to be combined with PS-KD, we also follow the hyperparameter values reported in the literature, for example, the hole size in Cutout is set to 8 and the parameter α of Beta distribution in CutMix is set to 1.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang in view of Kang to include wherein the at least one data augmentation technique includes a random permutation function, the random permutation function to adjust an image based on a beta distribution taught by Kim. The motivation for doing so would have been to produce randomly synthesized images as suggested by Kim (see Kim Introduction Page 6571 [right column paragraph 1]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Kang and Kim with Wang to obtain the invention specified in claim 6.
Regarding claim 13 (drawn to a method), claim 13 is rejected the same as claim 6 and the arguments similar to that presented above for claim 6 are equally applicable to the claim 13, and all the other limitations similar to claim 6 are not repeated herein, but incorporated by reference.
Regarding claim 18 (drawn to a method/non-transitory storage medium), claim 18 is rejected the same as claim 6 and the arguments similar to that presented above for claim 6 are equally applicable to the claim 18, and all the other limitations similar to claim 6 are not repeated herein, but incorporated by reference.
Claims 21 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. ("Memory-replay knowledge distillation." Sensors 21.8 (2021): 2792.) (hereinafter, Wang) in view of Tung et al. ("Similarity-preserving knowledge distillation." Proceedings of the IEEE/CVF international conference on computer vision. 2019.) (hereinafter, Tung), further in view of Kang et al. "Ensemble learning of lightweight deep learning models using knowledge distillation for image classification." Mathematics 8.10 (2020): 1652. (hereinafter, “Kang”), and further in view of Haralabopoulos et al. "Text data augmentations: Permutation, antonyms and negation." Expert Systems with Applications 177 (2021): 114769.. (hereinafter. “Haralabopoulos”).
Regarding claim 21, which claim 1 is incorporated, Wang fails to teach wherein the at least one data augmentation technique includes a permutation on logits of knowledge associated with the student neural network to generate the augmented student knowledge data.
Tang teaches wherein the at least one data augmentation technique [includes a permutation] on logits of knowledge associated with the student neural network to generate the augmented student knowledge data (Page 1369-1370 [right column last paragraph] “combine knowledge distillation with fine-tuning: we initialize the student network with source domain (in this case, ImageNet) pretrained weights, and then fine-tune the student to the target domain using both distillation and cross-entropy losses (Eq. 5)… We applied ImageNet-style data augmentation with horizontal flipping and random resized cropping during training. At test time, images were resized to 256x256 and center cropped to 224x224 for input to the network”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang to include Tang teaches wherein the at least one data augmentation technique [includes a permutation] on logits of knowledge associated with the student neural network to generate the augmented student knowledge data taught by Tung’s reference. The motivation for doing so would have been to transfer the knowledge of a network pre-trained on a large data-set to the new recognition task by fine-tuning as suggested by Tung (see Tung page 1369-1370 [right column paragraph 2]).
Further, Wang, Tang and Kang fail to teach wherein the data augmentation technique includes permutation.
Haralabopoulos teaches wherein the data augmentation technique includes permutation (Page 3 right column subsection 3.2.2 first paragraph “Our proposed permutation augmentation method is based on all the possible sentences that can be created from a predefined number of terms. For each sentence in the corpus we create extra sentences by randomly re-positioning all the terms.”.
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang in view of Tung, and further in view of Kang to include wherein the data augmentation technique includes permutation taught by Haralabopoulos’s reference. The motivation for doing so would have been to improve classification accuracy suggested by Haralabopoulos (see Haralabopoulos page 5 Subsection 4.1 first paragraph)).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tung, Kang and Haralabopoulos with Wang to obtain the invention specified in claim 21.
Regarding claim 22, which claim 21 is incorporated, Wang teaches wherein logits of knowledge associated with the teacher neural network are processed using mutual distillation in combination with the logits of knowledge associated with the student neural network [after the permutation] (Page 4, paragraph 2 “This improvement of MSD indicates that the self-distillation method can be regarded as a Deep Mutual Learning (DML) [16] method of four peers with different low-level weight sharing. We evaluate four-model DML directly and found comparable results”; further see algorithm 1 on page 8:)
However, Wang, Tung and Kang fail to teach permutation.
Haralabopoulos teaches permutation (Page 3 right column subsection 3.2.2 first paragraph “Our proposed permutation augmentation method is based on all the possible sentences that can be created from a predefined number of terms. For each sentence in the corpus we create extra sentences by randomly re-positioning all the terms.”.
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Wang in view of Tung, and further in view of Kang to include wherein the data augmentation technique includes permutation taught by Haralabopoulos’s reference. The motivation for doing so would have been to improve classification accuracy suggested by Haralabopoulos (see Haralabopoulos page 5 Subsection 4.1 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tung, Kang and Haralabopoulos with Wang to obtain the invention specified in claim 22.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Fukuda et al (US 11907845 B2) discloses a method of training teacher and student neural networks by jointly training lossless and lossy branches of a teacher network with shared weights.
Bagherinezhad et al. (US 11030486 B2) discloses a method of training neural networks by generating image crops from a dataset, and applying a neural network to produce output that are used as training labels.
Tung et al. (US 20200302295 A1) discloses a method of training a student neural network using knowledge distillation by generating student and teacher activation maps corresponding to layers and minimizing knowledge distillation loss.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights