DETAILED ACTION
This action is responsive to the amendment filed on 03/27/2026. Claims 1-16 and 18-21 are pending in the case. Claims 1, 9, 18, and 21 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/03/2026 is being considered by the examiner.
Claim Objections
Claim 2 is objected to because of the following informalities:
Claim 2, line 6, “with using a first resolution” should read “with a first resolution” as it appears to be a typographical error.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-16 and 18-21 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al., BRIDGENETS: STUDENT-TEACHER TRANSFER LEARNING BASED ON RECURSIVE NEURAL NETWORKS AND ITS APPLICATION TO DISTANT SPEECH RECOGNITION, 02/21/2018, https://arxiv.org/pdf/1710.10224, hereinafter referred to as “Kim” in view of Cai et al., U.S. Patent Application Publication No. 20230052483, hereinafter referred to as “Cai” in further view of Kim et al., U.S. Patent No. 11195093, hereinafter referred to as “Pat. Kim”.
Regarding claim 1, Kim teaches A deep learning training method implemented by a computing device (Kim, Page 3, Section 3.1, Paragraph 2, Lines 1-2, “Kaldi [20] and Microsoft Cognitive Toolkit (CNTK) [21] are used to train and decode BridgeNet”; A person of ordinary skill in the art would recognize “Kaldi” and “CNTK” require the use of a computing device), wherein the deep learning training method comprises:
obtaining a training set (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), a first neural network, and a second neural network (Kim, Page 2, Section 2.1, Lines 3-5, “Figure 1 presents a high-level block diagram of BridgeNet. Both student and teacher networks are constructed from a recursive network”; The student network is considered to be the “first neural network” and the teacher network is considered to be the “second neural network”), wherein the training set comprises a plurality of samples (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), wherein the first neural network comprises one or more first intermediate layers and a first quantity of shortcuts …, wherein each of the first intermediate layers comprises one or more first blocks without a first shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be an “intermediate layer” and thus the blocks labeled “I”, “F”, and “M” are considered to be the “one or more first blocks without a first shortcut connection” and the “residual LSTM layers” contain the “first quantity of shortcuts”), wherein the second neural network comprises a plurality of network layers, wherein the plurality of network layers comprises an output layer and one or more second intermediate layers, wherein each of the second intermediate layers comprises one or more second blocks with a second shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be a “network layer” and thus the “softmax” is considered to be the “output layer”, the blocks labeled “I”, “F”, and “M”, the “Knowledge Bridge[s]”, further, because the “Knowledge Bridge[s]” that output from the Teacher Network, they can also be considered “output layer[s]”, and the “residual LSTM layers” are considered to be the “second intermediate layers” and the “residual LSTM Layer[s]” are considered to be the “one or more second blocks with a second shortcut connection”), and wherein the first neural network and the second neural network include corresponding stages (Kim, Page 2, Figure 1; The “Teacher Network” squares, “Student Network” squares, and “Knowledge Bridge[s]” are considered to be the “corresponding stages”) ; and
performing, based on the training set, at least one time of iterative training on the first neural network to obtain a trained first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; A person of ordinary skill in the art would recognize that minimizing a loss function is performed iteratively),
wherein the at least one time of iterative training comprises: using a first output of at least one of the first intermediate layers as a first input of at least one of the network layers to obtain a first output result of the at least one of the network layers (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The “feature representation
q
i
” is considered to be the “first output” and is used as an input to the “knowledge bridge” which is part of “one of the network layers”; and the cross-entropy loss is considered to be the “first output result”), wherein using the first output comprises using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 2, Figure 1; see also Kim, Page 3, Figure 3; The “Student Network” squares of Figure 1 are considered to be the “first stage of the first neural network”; the “knowledge bridges” in Figures 1 and 3 and “softmax” layer in Figure 3 are considered to be the “second stage of the second neural network” and the “knowledge bridges” of Figure 1 are considered to be the “output layer” and
P
S
(
x
t
;
ϕ
s
)
is considered to be “a first prediction label”);
outputting, in a second stage of the second neural network, the first output result and a second output result, wherein the second output result is based on output of a first stage of the second neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 3, Figure 3; The prediction output is considered to be the second output result; and because the prediction is determined based on training the model to minimize the error, which is determined from the feature representations/hints of the “Teacher Network” squares which are considered to be the “output of a first stage of the second neural network”, the second output result is considered to be “based on output of a first stage of the second neural network”; and
updating, according to a first loss function, the first neural network to obtain an updated first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions,
L
ϕ
s
=
∑
i
=
1
N
α
i
e
i
(
ϕ
s
)
”; Equation 3 is considered to be the “first loss function”),
wherein the first loss function comprises a first constraint term based on the first output result … (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “first constraint term”)…
Kim does not explicitly teach the quantity of shortcuts of the first neural network being based on a memory size of the computing device nor the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Cai teaches the quantity of shortcuts of the first neural network being based on a memory size of the computing device (Cai, Paragraph 0029, Lines 11-15, “In addition, in various examples, the number of residual blocks and feature dimensions may be pruned in order to achieve real time performance with limited computation capability and memory bandwidth in mobile platform”; The “shortcuts” are only present in the residual blocks, thus adjusting the number of residual blocks based on “memory bandwidth” is considered to be a “first quantity of shortcuts that is based on a memory size of the computing device”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention to have modified the deep learning training method of Kim to include adjusting a quantity of shortcuts in the first neural network based on the memory size of the computing device as taught by Cai. The motivation to do so would have been to preserve performance of the deep learning method regardless of the computing device the model is used on (Cai, Paragraph 0029).
The proposed combination does not explicitly teach the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Pat. Kim teaches the first loss function comprises a second constraint term, and wherein the second constraint term comprises a loss value corresponding to the first prediction label (Pat. Kim, Col 4, Lines 47-57, “If
x
t
is an input feature for both teacher and student networks at time t, then
P
T
(
x
t
)
is a softmax output of a teacher network and
P
S
(
x
t
)
is a softmax output of a student network. The student network is, in turn, trained to minimize a weighted average of two objective functions as in Equation (4):
L
K
D
φ
=
1
-
α
∑
t
=
1
T
C
E
P
T
x
t
,
P
S
x
t
;
φ
+
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
(4) where
φ
is a collection of parameters in a student network and
L
K
D
φ
is a loss function for KD”; The second term, “
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
” is considered to be the “second constraint term” and “
P
S
x
t
;
φ
” is considered to be the “first prediction label”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the deep learning method of the proposed combination to include the loss term having a second constraint term as taught by Pat. Kim. The motivation for doing so would have been to allow the student neural network to learn to generalize similarly to the teacher network while preserving accuracy (Pat. Kim, Col 4, Lines 27-32, “Knowledge distillation (KD) transfers generalization ability of a bigger teacher network to a typically much smaller student network. It provides soft-target information computed by the teacher network, in addition to its hard-targets, so the student network can learn to generalize similarly”).
Regarding claim 2, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein the at least one time of iterative training comprises the first neural network and the second neural network each processing input (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-4, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels. A teacher network is trained with
x
t
*
and
y
t
pairs”; see also Kim, Page 2, Figure 1).
The proposed combination thus far does not explicitly teach that input being images with using a first resolution.
Cai teaches input being images with using a first resolution (Cai, Paragraph 0024, Lines 1-2, “The CNN 104 may be any upscaling framework that takes low resolution frames 102 as input”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention to have modified the deep learning training method of the proposed combination to include taking images as input. The motivation to do so would have been the ability to restore low resolution images (Cai, Paragraph 0014, Lines 1-3, “Deep learning based super resolution may be used in restoring low resolution images and video frames to high resolution images and video frames”).
Regarding claim 3, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein using the first output further comprises using a third output of a last intermediate layer in the first neural network to obtain the first prediction label (Kim, Page 2, Figure 1, “Knowledge Bridge i”; see also Kim, Page 3, Figure 3, “Knowledge Bridge:LSTM3”; A person of ordinary skill in the art can recognize this Knowledge Bridge after the “L” block and referring back to Figure 1, it is placed in the final “recursion” which is considered to be the “last intermediate layer”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The “knowledge bridge” is considered to be the “output layer” and
P
S
(
x
t
;
ϕ
s
)
is considered to be “the first prediction label”).
Regarding claim 4, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein the first constraint term comprises a first loss value between the first prediction label and a true label of a sample input to the first neural network (Kim, Page 2, Section 2.1, Paragraph 2, Lines 3-5, “A teacher network is trained with
x
t
*
and
y
t
pairs. The trained teacher network provides its internal feature representations as hints to a student network”; Kim, Page 2, Section 2.1, Paragraph 2, Line 2, “
x
t
*
is enhanced or less noisy data”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “first constraint term”, “
P
S
(
x
t
;
ϕ
s
)
” is considered to be the “first prediction label” and “
P
T
x
t
*
;
ϕ
T
T
is considered to be “a true label” because the teacher model only provides the hints once the teacher is fully trained, so the teacher model outputs are considered true labels).
Regarding claim 5, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein using the first output comprises:
obtaining the first output (Kim, Page 2, Figure 1, “Knowledge Bridge i”); and
using the first output as a second input of at least one of the second intermediate layers to obtain a second output of the at least one of the second intermediate layers (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); While evaluating the second knowledge bridge, “
P
S
(
x
t
;
ϕ
s
)
” is considered to be the “second output” and the knowledge bridge is considered to be the “at least one of the second intermediate layers”),
wherein updating the first neural network comprises: using a sample input to the first neural network as a second input of the second neural network to obtain a third output of the at least one of the second intermediate layers (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); “
x
t
*
” is considered to be the “sample input” that is also “a second input of the second neural network” and “
P
T
x
t
*
;
ϕ
T
T
is considered to be the “third output” when evaluating the second knowledge bridge); and
updating, according to the first loss function, the first neural network from previous iterative training to obtain the first neural network in current iterative training (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions,
L
ϕ
s
=
∑
i
=
1
N
α
i
e
i
(
ϕ
s
)
”; Equation 3 is considered to be the “first loss function”),
wherein the first loss function further comprises a second constraint term, and wherein the second constraint term comprises a loss value between the second output and the third output (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “second constraint term” while evaluating the second knowledge bridge, “
P
S
(
x
t
;
ϕ
s
)
” is considered to be the “second output” and “
P
T
x
t
*
;
ϕ
T
T
is considered to be the “third output”).
Regarding claim 6, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein the at least one time of iterative training further comprises:
obtaining a prediction label of the second neural network for a sample input to the first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
(2)”; “
P
T
x
t
*
;
ϕ
T
T
” is considered to be the “prediction label of the second neural network” and “
x
t
*
” is the “sample input to the first neural network”);
…
updating, based on a loss value, a parameter of the second neural network to obtain the second neural network in current iterative training (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-4, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels. A teacher network is trained with
x
t
*
and
y
t
pairs” A person of ordinary skill in the art would recognize that training a model requires updating the model based on a loss value).
The proposed combination does not explicitly teach calculating a loss value, based on the prediction label and a true label of the sample input to the first neural network.
Pat. Kim teaches calculating a loss value, based on the prediction label and a true label of the sample input to the first neural network (Pat. Kim, Col 4, Lines 47-57, “If
x
t
is an input feature for both teacher and student networks at time t, then
P
T
(
x
t
)
is a softmax output of a teacher network and
P
S
(
x
t
)
is a softmax output of a student network. The student network is, in turn, trained to minimize a weighted average of two objective functions as in Equation (4):
L
K
D
φ
=
1
-
α
∑
t
=
1
T
C
E
P
T
x
t
,
P
S
x
t
;
φ
+
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
(4) where
φ
is a collection of parameters in a student network and
L
K
D
φ
is a loss function for KD”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the deep learning method of the proposed combination to include calculating a loss term as taught by Pat. Kim. The motivation for doing so would have been to allow the student neural network to learn to generalize similarly to the teacher network while preserving accuracy (Pat. Kim, Col 4, Lines 27-32, “Knowledge distillation (KD) transfers generalization ability of a bigger teacher network to a typically much smaller student network. It provides soft-target information computed by the teacher network, in addition to its hard-targets, so the student network can learn to generalize similarly”).
Regarding claim 7, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein before performing the at least one time of iterative training, the deep learning training method further comprises updating, based on the training set, a parameter of the second neural network to obtain an updated second neural network (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-5, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels. A teacher network is trained with
x
t
*
and
y
t
pairs. The trained teacher network provides its internal feature representations as hints to a student network”).
Regarding claim 8, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein the first neural network is for a classification task (Kim, Page 1, Abstract, Lines 4-6, “we introduce novel student-teacher transfer learning, BridgeNet which can provide a solution to improve distant speech recognition; Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”; Kim, Page 3, Figure 3, “SoftMax”; The model ultimately uses labels and a softmax layer to perform speech recognition as a “classification task”).
Regarding claim 9, Kim teaches A training apparatus, comprising: a memory configured to store instructions; and a processor coupled to the memory and configured to execute the instructions (Kim, Page 3, Section 3.1, Paragraph 2, Lines 1-2, “Kaldi [20] and Microsoft Cognitive Toolkit (CNTK) [21] are used to train and decode BridgeNet”; A person of ordinary skill in the art would recognize “Kaldi” and “CNTK” require the use of a generic computer, providing evidence for a memory, instructions, and a processor) to:
obtain a training set (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), a first neural network, and a second neural network (Kim, Page 2, Section 2.1, Lines 3-5, “Figure 1 presents a high-level block diagram of BridgeNet. Both student and teacher networks are constructed from a recursive network”; The student network is considered to be the “first neural network” and the teacher network is considered to be the “second neural network”), wherein the training set comprises a plurality of samples (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), wherein the first neural network comprises one or more first intermediate layers and a first quantity of shortcuts …, wherein each of the first intermediate layers comprises one or more first blocks without a first shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be an “intermediate layer” and thus the blocks labeled “I”, “F”, and “M” are considered to be the “one or more first blocks without a first shortcut connection” and the “residual LSTM layers” contain the “first quantity of shortcuts”), wherein the second neural network comprises a plurality of network layers, wherein the plurality of network layers comprises an output layer and one or more second intermediate layers, wherein each of the second intermediate layers comprises one or more second blocks with a second shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be a “network layer” and thus the “softmax” is considered to be the “output layer”, the blocks labeled “I”, “F”, and “M”, the “Knowledge Bridge[s]”, further, because the “Knowledge Bridge[s]” that output from the Teacher Network, they can also be considered “output layer[s]”, and the “residual LSTM layers” are considered to be the “second intermediate layers” and the “residual LSTM Layer[s]” are considered to be the “one or more second blocks with a second shortcut connection”), and wherein the first neural network and the second neural network include corresponding stages (Kim, Page 2, Figure 1; The “Teacher Network” squares, “Student Network” squares, and “Knowledge Bridge[s]” are considered to be the “corresponding stages”) ; and
perform, based on the training set, at least one time of iterative training on the first neural network to obtain a trained first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; A person of ordinary skill in the art would recognize that minimizing a loss function is performed iteratively),
wherein the at least one time of iterative training comprises: using a first output of at least one of the first intermediate layers as a first input of at least one of the network layers to obtain a first output result of the at least one of the network layers (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The “feature representation
q
i
” is considered to be the “first output” and is used as an input to the “knowledge bridge” which is part of “one of the network layers”; and the cross-entropy loss is considered to be the “first output result”), wherein using the first output comprises using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 2, Figure 1; see also Kim, Page 3, Figure 3; The “Student Network” squares of Figure 1 are considered to be the “first stage of the first neural network”; the “knowledge bridges” in Figures 1 and 3 and “softmax” layer in Figure 3 are considered to be the “second stage of the second neural network” and the “knowledge bridges” of Figure 1 are considered to be the “output layer” and
P
S
(
x
t
;
ϕ
s
)
is considered to be “a first prediction label”);
outputting, in a second stage of the second neural network, the first output result and a second output result, wherein the second output result is based on output of a first stage of the second neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 3, Figure 3; The prediction output is considered to be the second output result; and because the prediction is determined based on training the model to minimize the error, which is determined from the feature representations/hints of the “Teacher Network” squares which are considered to be the “output of a first stage of the second neural network”, the second output result is considered to be “based on output of a first stage of the second neural network”; and
updating, according to a first loss function, the first neural network to obtain an updated first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions,
L
ϕ
s
=
∑
i
=
1
N
α
i
e
i
(
ϕ
s
)
”; Equation 3 is considered to be the “first loss function”),
wherein the first loss function comprises a first constraint term based on the first output result … (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “first constraint term”)…
Kim does not explicitly teach the quantity of shortcuts of the first neural network being based on a memory size of the training apparatus nor the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Cai teaches the quantity of shortcuts of the first neural network being based on a memory size of the training apparatus (Cai, Paragraph 0029, Lines 11-15, “In addition, in various examples, the number of residual blocks and feature dimensions may be pruned in order to achieve real time performance with limited computation capability and memory bandwidth in mobile platform”; The “shortcuts” are only present in the residual blocks, thus adjusting the number of residual blocks based on “memory bandwidth” is considered to be a “first quantity of shortcuts that is based on a memory size of the computing device”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention to have modified the deep learning training method of Kim to include adjusting a quantity of shortcuts in the first neural network based on the memory size of the computing device as taught by Cai. The motivation to do so would have been to preserve performance of the deep learning method regardless of the computing device the model is used on (Cai, Paragraph 0029).
The proposed combination does not explicitly teach the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Pat. Kim teaches the first loss function comprises a second constraint term, and wherein the second constraint term comprises a loss value corresponding to the first prediction label (Pat. Kim, Col 4, Lines 47-57, “If
x
t
is an input feature for both teacher and student networks at time t, then
P
T
(
x
t
)
is a softmax output of a teacher network and
P
S
(
x
t
)
is a softmax output of a student network. The student network is, in turn, trained to minimize a weighted average of two objective functions as in Equation (4):
L
K
D
φ
=
1
-
α
∑
t
=
1
T
C
E
P
T
x
t
,
P
S
x
t
;
φ
+
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
(4) where
φ
is a collection of parameters in a student network and
L
K
D
φ
is a loss function for KD”; The second term, “
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
” is considered to be the “second constraint term” and “
P
S
x
t
;
φ
” is considered to be the “first prediction label”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the deep learning method of the proposed combination to include the loss term having a second constraint term as taught by Pat. Kim. The motivation for doing so would have been to allow the student neural network to learn to generalize similarly to the teacher network while preserving accuracy (Pat. Kim, Col 4, Lines 27-32, “Knowledge distillation (KD) transfers generalization ability of a bigger teacher network to a typically much smaller student network. It provides soft-target information computed by the teacher network, in addition to its hard-targets, so the student network can learn to generalize
similarly”).
Regarding claim 10, the rejection of claim 9 is incorporated, and further, the proposed combination teaches wherein updating the first neural network comprises using output of one or more network layers in the second neural network and a first preset label output by the second neural network as a constraint (Kim, Page 2, Section 2.1, Paragraph 3, Lines 5-6 and Equation 2, “Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
(2)”; “
h
i
” is considered to be the “output of one or more network layers in the second neural network” and “
P
T
x
t
*
;
ϕ
T
” is considered to be the “first preset label”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions,
L
ϕ
s
=
∑
i
=
1
N
α
i
e
i
(
ϕ
s
)
”; Equation 3 is considered to be the “first loss function”)
Regarding claim 11, the rejection of claim 10 is incorporated, and further, claim 11 is substantially similar to claim 3 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 12, the rejection of claim 10 is incorporated, and further, wherein the first constraint term comprises a second loss value between the first prediction label and a second prediction label, and the second prediction label is a third output result of the second neural network for a sample input (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “first constraint term”, “
P
S
(
x
t
;
ϕ
s
)
” is considered to be the “first prediction label” and “
P
T
x
t
*
;
ϕ
T
T
is considered to be the “second prediction label”).
Regarding claim 13, the rejection of claim 9 is incorporated, and further, claim 13 is substantially similar to claim 5 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 14, the rejection of claim 9 is incorporated, and further, claim 14 is substantially similar to claim 6 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 15, the rejection of claim 9 is incorporated, and further, claim 15 is substantially similar to claim 7 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 16, the rejection of claim 9 is incorporated, and further, claim 16 is substantially similar to claim 8 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 18, Kim teaches A computer program product comprising instructions stored on a non-transitory computer-readable medium that, when executed by a processor, cause a computing device (Kim, Page 3, Section 3.1, Paragraph 2, Lines 1-2, “Kaldi [20] and Microsoft Cognitive Toolkit (CNTK) [21] are used to train and decode BridgeNet”; A person of ordinary skill in the art would recognize “Kaldi” and “CNTK” require the use of a generic computer, providing evidence for a non-transitory computer-readable medium, instructions, and a processor) to:
obtain a training set (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), a first neural network, and a second neural network (Kim, Page 2, Section 2.1, Lines 3-5, “Figure 1 presents a high-level block diagram of BridgeNet. Both student and teacher networks are constructed from a recursive network”; The student network is considered to be the “first neural network” and the teacher network is considered to be the “second neural network”), wherein the training set comprises a plurality of samples (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), wherein the first neural network comprises one or more first intermediate layers and a first quantity of shortcuts …, wherein each of the first intermediate layers comprises one or more first blocks without a first shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be an “intermediate layer” and thus the blocks labeled “I”, “F”, and “M” are considered to be the “one or more first blocks without a first shortcut connection” and the “residual LSTM layers” contain the “first quantity of shortcuts”), wherein the second neural network comprises a plurality of network layers, wherein the plurality of network layers comprises an output layer and one or more second intermediate layers, wherein each of the second intermediate layers comprises one or more second blocks with a second shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be a “network layer” and thus the “softmax” is considered to be the “output layer”, the blocks labeled “I”, “F”, and “M”, the “Knowledge Bridge[s]”, further, because the “Knowledge Bridge[s]” that output from the Teacher Network, they can also be considered “output layer[s]”, and the “residual LSTM layers” are considered to be the “second intermediate layers” and the “residual LSTM Layer[s]” are considered to be the “one or more second blocks with a second shortcut connection”), and wherein the first neural network and the second neural network include corresponding stages (Kim, Page 2, Figure 1; The “Teacher Network” squares, “Student Network” squares, and “Knowledge Bridge[s]” are considered to be the “corresponding stages”) ; and
perform, based on the training set, at least one time of iterative training on the first neural network to obtain a trained first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; A person of ordinary skill in the art would recognize that minimizing a loss function is performed iteratively),
wherein the at least one time of iterative training comprises: using a first output of at least one of the first intermediate layers as a first input of at least one of the network layers to obtain a first output result of the at least one of the network layers (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The “feature representation
q
i
” is considered to be the “first output” and is used as an input to the “knowledge bridge” which is part of “one of the network layers”; and the cross-entropy loss is considered to be the “first output result”), wherein using the first output comprises using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 2, Figure 1; see also Kim, Page 3, Figure 3; The “Student Network” squares of Figure 1 are considered to be the “first stage of the first neural network”; the “knowledge bridges” in Figures 1 and 3 and “softmax” layer in Figure 3 are considered to be the “second stage of the second neural network” and the “knowledge bridges” of Figure 1 are considered to be the “output layer” and
P
S
(
x
t
;
ϕ
s
)
is considered to be “a first prediction label”);
outputting, in a second stage of the second neural network, the first output result and a second output result, wherein the second output result is based on output of a first stage of the second neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 3, Figure 3; The prediction output is considered to be the second output result; and because the prediction is determined based on training the model to minimize the error, which is determined from the feature representations/hints of the “Teacher Network” squares which are considered to be the “output of a first stage of the second neural network”, the second output result is considered to be “based on output of a first stage of the second neural network”; and
updating, according to a first loss function, the first neural network to obtain an updated first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions,
L
ϕ
s
=
∑
i
=
1
N
α
i
e
i
(
ϕ
s
)
”; Equation 3 is considered to be the “first loss function”),
wherein the first loss function comprises a first constraint term based on the first output result … (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “first constraint term”)…
Kim does not explicitly teach the quantity of shortcuts of the first neural network being based on a memory size of the computing device nor the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Cai teaches the quantity of shortcuts of the first neural network being based on a memory size of the computing device (Cai, Paragraph 0029, Lines 11-15, “In addition, in various examples, the number of residual blocks and feature dimensions may be pruned in order to achieve real time performance with limited computation capability and memory bandwidth in mobile platform”; The “shortcuts” are only present in the residual blocks, thus adjusting the number of residual blocks based on “memory bandwidth” is considered to be a “first quantity of shortcuts that is based on a memory size of the computing device”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention to have modified the deep learning training method of Kim to include adjusting a quantity of shortcuts in the first neural network based on the memory size of the computing device as taught by Cai. The motivation to do so would have been to preserve performance of the deep learning method regardless of the computing device the model is used on (Cai, Paragraph 0029).
The proposed combination does not explicitly teach the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Pat. Kim teaches the first loss function comprises a second constraint term, and wherein the second constraint term comprises a loss value corresponding to the first prediction label (Pat. Kim, Col 4, Lines 47-57, “If
x
t
is an input feature for both teacher and student networks at time t, then
P
T
(
x
t
)
is a softmax output of a teacher network and
P
S
(
x
t
)
is a softmax output of a student network. The student network is, in turn, trained to minimize a weighted average of two objective functions as in Equation (4):
L
K
D
φ
=
1
-
α
∑
t
=
1
T
C
E
P
T
x
t
,
P
S
x
t
;
φ
+
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
(4) where
φ
is a collection of parameters in a student network and
L
K
D
φ
is a loss function for KD”; The second term, “
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
” is considered to be the “second constraint term” and “
P
S
x
t
;
φ
” is considered to be the “first prediction label”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the deep learning method of the proposed combination to include the loss term having a second constraint term as taught by Pat. Kim. The motivation for doing so would have been to allow the student neural network to learn to generalize similarly to the teacher network while preserving accuracy (Pat. Kim, Col 4, Lines 27-32, “Knowledge distillation (KD) transfers generalization ability of a bigger teacher network to a typically much smaller student network. It provides soft-target information computed by the teacher network, in addition to its hard-targets, so the student network can learn to generalize similarly”).
Regarding claim 19, the rejection of claim 18 is incorporated, and further, claim 19 is substantially similar to claim 2 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 20, the rejection of claim 19 is incorporated, and further, claim 20 is substantially similar to claim 3 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 21, Kim teaches A method comprising:
obtaining a training set (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), a first neural network, and a second neural network (Kim, Page 2, Section 2.1, Lines 3-5, “Figure 1 presents a high-level block diagram of BridgeNet. Both student and teacher networks are constructed from a recursive network”; The student network is considered to be the “first neural network” and the teacher network is considered to be the “second neural network”), wherein the training set comprises a plurality of samples (Kim, Page 2, Section 2.1, Paragraph 2, Lines 1-3, “BridgeNet uses a collection of triplets as training data:
(
x
t
*
,
x
t
,
y
t
)
.
x
t
*
is enhanced or less noisy data,
x
t
and
y
t
are noisy data and their labels”), wherein the first neural network comprises one or more first intermediate layers and a first quantity of shortcuts …, wherein each of the first intermediate layers comprises one or more first blocks without a first shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be an “intermediate layer” and thus the blocks labeled “I”, “F”, and “M” are considered to be the “one or more first blocks without a first shortcut connection” and the “residual LSTM layers” contain the “first quantity of shortcuts”), wherein the second neural network comprises a plurality of network layers, wherein the plurality of network layers comprises an output layer and one or more second intermediate layers, and wherein each of the second intermediate layers comprises one or more second blocks with a second shortcut connection (Kim, Page 3, Col 1, Paragraph 2, Lines 1-4, “Figure 3 shows how the Bridgenet concept is applied to the recursive network of Figure 2. It has four components: CNN layers (I), first LSTM layers (F), second LSTM layers (L) and dimension reduction layer (M)”; see also Kim, Page 3, Figures 2 and 3; Each “recursion” is considered to be a “network layer” and thus the “softmax” is considered to be the “output layer”, the blocks labeled “I”, “F”, and “M”, the “Knowledge Bridge[s]”, further, because the “Knowledge Bridge[s]” that output from the Teacher Network, they can also be considered “output layer[s]”, and the “residual LSTM layers” are considered to be the “second intermediate layers” and the “residual LSTM Layer[s]” are considered to be the “one or more second blocks with a second shortcut connection”); and
performing, based on the training set, at least one time of iterative training on the first neural network to obtain a trained first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; A person of ordinary skill in the art would recognize that minimizing a loss function is performed iteratively),
wherein the at least one time of iterative training comprises: using a first output of at least one of the first intermediate layers as a first input of at least one of the network layers to obtain a first output result of the at least one of the network layers (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The “feature representation
q
i
” is considered to be the “first output” and is used as an input to the “knowledge bridge” which is part of “one of the network layers”; and the cross-entropy loss is considered to be the “first output result”), wherein using the first output comprises using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 2, Figure 1; see also Kim, Page 3, Figure 3; The “Student Network” squares of Figure 1 are considered to be the “first stage of the first neural network”; the “knowledge bridges” in Figures 1 and 3 and “softmax” layer in Figure 3 are considered to be the “second stage of the second neural network” and the “knowledge bridges” of Figure 1 are considered to be the “output layer” and
P
S
(
x
t
;
ϕ
s
)
is considered to be “a first prediction label”);
outputting, in a second stage of the second neural network, the first output result and a second output result, wherein the second output result is based on output of a first stage of the second neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); see also Kim, Page 3, Figure 3; The prediction output is considered to be the second output result; and because the prediction is determined based on training the model to minimize the error, which is determined from the feature representations/hints of the “Teacher Network” squares which are considered to be the “output of a first stage of the second neural network”, the second output result is considered to be “based on output of a first stage of the second neural network”); and
updating, according to a first loss function, the first neural network to obtain an updated first neural network (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions,
L
ϕ
s
=
∑
i
=
1
N
α
i
e
i
(
ϕ
s
)
”; Equation 3 is considered to be the “first loss function”),
wherein the first loss function comprises a first constraint term based on the first output result … (Kim, Page 2, Section 2.1, Paragraph 3, Lines 7-9, “The parameters of the student network are then optimized by minimizing a weighted sum of all corresponding loss functions”; Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2, “An error measure
e
i
of how a feature representation
q
i
from a student network agrees with the hint
h
i
is computed at the knowledge bridge as a MSE loss,
e
i
ϕ
s
=
∑
t
=
1
L
h
i
x
t
*
-
q
i
(
x
t
;
ϕ
s
)
2
(1) where
ϕ
s
is the learnable parameters of a student network. Since
h
i
and
q
i
are softmax probabilities of teacher and student networks, the cross-entropy loss is used for
e
1
instead.
e
1
ϕ
s
=
-
∑
t
=
1
L
P
T
x
t
*
;
ϕ
T
T
l
o
g
P
S
(
x
t
;
ϕ
s
)
” (2); The
e
1
loss is considered to be the “first constraint term”)…
Kim does not explicitly teach the quantity of shortcuts of the first neural network being based on a memory size nor the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Cai teaches the quantity of shortcuts of the first neural network being based on a memory size (Cai, Paragraph 0029, Lines 11-15, “In addition, in various examples, the number of residual blocks and feature dimensions may be pruned in order to achieve real time performance with limited computation capability and memory bandwidth in mobile platform”; The “shortcuts” are only present in the residual blocks, thus adjusting the number of residual blocks based on “memory bandwidth” is considered to be a “first quantity of shortcuts that is based on a memory size of the computing device”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention to have modified the deep learning training method of Kim to include adjusting a quantity of shortcuts in the first neural network based on the memory size of the computing device as taught by Cai. The motivation to do so would have been to preserve performance of the deep learning method regardless of the computing device the model is used on (Cai, Paragraph 0029).
The proposed combination does not explicitly teach the loss function having a second constraint term … wherein the second constraint term comprises a loss value corresponding to the first prediction label.
Pat. Kim teaches the first loss function comprises a second constraint term, and wherein the second constraint term comprises a loss value corresponding to the first prediction label (Pat. Kim, Col 4, Lines 47-57, “If
x
t
is an input feature for both teacher and student networks at time t, then
P
T
(
x
t
)
is a softmax output of a teacher network and
P
S
(
x
t
)
is a softmax output of a student network. The student network is, in turn, trained to minimize a weighted average of two objective functions as in Equation (4):
L
K
D
φ
=
1
-
α
∑
t
=
1
T
C
E
P
T
x
t
,
P
S
x
t
;
φ
+
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
(4) where
φ
is a collection of parameters in a student network and
L
K
D
φ
is a loss function for KD”; The second term, “
α
∑
t
=
1
T
C
E
(
y
t
l
a
b
e
l
,
P
S
x
t
;
φ
)
” is considered to be the “second constraint term” and “
P
S
x
t
;
φ
” is considered to be the “first prediction label”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the deep learning method of the proposed combination to include the loss term having a second constraint term as taught by Pat. Kim. The motivation for doing so would have been to allow the student neural network to learn to generalize similarly to the teacher network while preserving accuracy (Pat. Kim, Col 4, Lines 27-32, “Knowledge distillation (KD) transfers generalization ability of a bigger teacher network to a typically much smaller student network. It provides soft-target information computed by the teacher network, in addition to its hard-targets, so the student network can learn to generalize similarly”).
Response to Arguments
Applicant’s amendments to the claims with respect to 35 U.S.C. 112(b) indefiniteness rejections to the claims have been fully considered, and overcome the rejections set forth in the nonfinal office action dated 12/31/2025. Consequently, the objections to the claims have been withdrawn.
Applicant’s arguments regarding the 35 U.S.C. 103 rejections of the claims have been fully considered but are unpersuasive.
Applicant first argues, on page 15, paragraph 2 of the response, that the combination of Kim, Cai, and Pat. Kim fails to disclose (1) the first neural network and the second neural network include corresponding stages, using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer, and outputting, in a second stage of the second neural network, the first output result and a second output result, wherein the second output result is based on output of a first stage of the second neural network, and (2) a first loss function comprising a first constraint term based on the first output result and a second constraint term, wherein the second constraint term comprises a loss value corresponding to the first prediction label. Examiner respectfully disagrees. Applicant’s specification paragraph 0083 discloses that each layer is referred to as a stage. Therefore, the broadest reasonable interpretation of “stage” includes that which each stage is simply a layer in a neural network and Kim teaches that both neural networks include multiple layers. Kim also teaches output of both the teacher and student neural network being passed to the knowledge bridges to obtain prediction labels, which is considered to be “using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer”. Kim also discloses outputting output results and a loss function with a first constraint term. Pat. Kim teaches a loss function with multiple constraint terms, and discloses a constraint term which comprises a loss value corresponding to the first prediction label. The combination of these references discloses everything argued by the applicant. For a more in depth analysis of each limitation, see the updated 35 U.S.C. 103 rejection seen above.
Applicant next argues, in page 16, final paragraph – page 18, final paragraph of the response, that Kim fails to teach using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer. Examiner respectfully disagrees. The “knowledge bridges” of Kim are considered to be network layers of both neural networks, as shown in Kim, Figure 1. Therefore, because the feature representations of the student network (first neural network) are output to the knowledge bridge and used to obtain a first prediction label (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2; see also Kim, Page 2, Figure 1; see also Kim, Page 3, Figure 3); Kim teaches using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer. The broadest reasonable interpretation of “stage” includes Kim’s knowledge bridges, as the claim places no limitations on the interpretation of “stage”.
Applicant next argues, on page 18, final paragraph of the response, that Kim fails to teach outputting, in a second stage of the second neural network, the first output result and a second output result, wherein the second output result is based on output of a first stage of the second neural network. Examiner respectfully disagrees. Applicant has placed no limitations on the “stages” limitations; thus, the broadest reasonable interpretation of “stages” includes that which multiple layers are incorporated in a single stage. Thus, the “softmax” layer of Kim is considered to be part of the second stage, and layers of the second stage output “an error measure” as well as the “prediction output” (Kim, Page 2, Section 2.1, Paragraph 3, Lines 1-6 and Equations 1 and 2; see also Kim, Page 3, Figure 3). For a more in depth explanation, see the updated 35 U.S.C. 103 rejection above.
Applicant next argues, in page 18, final three lines of the response, that Kim does not teach a first loss function comprises a first constraint term based on the first output result and a second constraint term, and wherein the second constraint term comprises a loss value corresponding to the first prediction label. Examiner makes no such assertion. Kim does teach a loss function comprising a first constraint term based on the first output result. Pat. Kim teaches a loss function with multiple constraint terms, and discloses a constraint term which comprises a loss value corresponding to the first prediction label. The combination of these references discloses the loss function.
Applicant next argues, on page 19, final paragraph – page 20 of the response, that Pat. Kim does not teach using output of a first stage of the first neural network as input to a second stage of the second neural network to obtain a first prediction label of the output layer and thus, Pat. Kim does not teach a first loss function comprises a first constraint term based on the first output result and a second constraint term, and wherein the second constraint term comprises a loss value corresponding to the first prediction label. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Further, Kim teaches a loss function comprising a first constraint term based on the first output result. Pat. Kim teaches a loss function with multiple constraint terms, and discloses a constraint term which comprises a loss value corresponding to the first prediction label. The combination of these references discloses the loss function. Applicant has made no argument regarding the references’ combination.
Applicant's arguments regarding the remainder of the claims rely upon the arguments asserted with respect to the independent claims, and are thus unpersuasive.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOLLY CLARKE SIPPEL whose telephone number is (571)272-3270. The examiner can normally be reached Monday - Friday, 7:30 a.m. - 4:30 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571)272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.C.S./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights