DETAILED ACTION
This action is in response to the application filed 09/13/2024. Claims 1-20 are pending and have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/23/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
[0071]: “The compression of the kernel matrix inherently increase the entropy” seems to be a typo of “The compression of the kernel matrix inherently increases the entropy”
[0106]: In “where one-
a
i
,
a
j
, are one-dimensional vectors”, it’s unclear what “one-
a
i
” is intended to mean
Appropriate correction is required.
Drawings
The drawings are objected to because Figures 6-7 and 9-10 are blurry and difficult to parse.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 1 is objected to because of the following informalities: “on the basis the sampled training data labels” in limitation 10 seems to be a typo of “on the basis of the sampled training data labels”. Appropriate correction is required.
Claim 13 is objected to because of the following informalities “wherein selecting the tunable hyperparameter selecting the hyperparameter using trial and error” seems to be a typo of “wherein selecting the tunable hyperparameter comprises using trial and error”.
Claim 15 is objected to because of the following informalities: “on the basis the sampled training data labels” in limitation 12 seems to be a typo of “on the basis of the sampled training data labels”. Appropriate correction is required.
Claim 20 is objected to because of the following informalities: “receiving, for a local computing node, an input kernel matrix and an label kernel matrix” in limitation 5 seems to be a typo of “receiving, for a local computing node, an input kernel matrix and a label kernel matrix”.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites "updating the parameters of the output layer on the basis of a loss computed for the output layer on the basis the sampled training data labels and the predicted labels" in its 10th limitation. There is insufficient antecedent basis for “the sampled training data labels” in the claim. Thus, the scope of the claim is rendered indefinite. This deficiency is present in substantially similar independent claim 15, and inherited by all dependent claims. “the sampled training data labels” is interpreted as comprising the sampled ground truth data samples.
Claim 20 recites “receiving, from the input computing node, an input activation map” in its 6th limitation, and “transmitting, to the output computing node, the first intermediate layer” in its final limitation. There is insufficient antecedent basis for “the input computing node” and “the output computing node”. Thus, the scope of the claim is rendered indefinite. “the input computing node” and “the output computing node” are interpreted as referring to the “input edge computing node” and “output edge computing node”, respectively, of limitation 4.
Additionally, claim 20 recites “performing computations of the first intermediate layer, on the basis of … parameters of the input first intermediate layer” in its 7th limitation. There is insufficient antecedent basis for “the input first intermediate layer”. There is insufficient antecedent basis for “the output computing node”. Thus, the scope of the claim is rendered indefinite. Based on similar limitations of independent claims 1 and 15, “the input first intermediate layer” is interpreted as referring to “the first intermediate layer”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Palanisamy (SplitEasy: A Practical Approach for Training ML models on Mobile Devices, published 1/29/2021, arXiv:2011.04232v2) in view of Ma (The HSIC Bottleneck: Deep Learning without Back-Propagation, published 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) pp. 5085-5092).
Regarding claim 1, Palanisamy discloses [a] method at a local computing node: “SplitEasy can function between any number of devices, but for this paper, we will stick with a basic client-server architecture as shown in Figure 2. The client (local computing node) is a mobile device that acts as both the starting and the ending point. That is, at any point of time, there are two separate instances running in parallel on the mobile device. We denote the start point as model A and the endpoint as model C. We use a single stand-alone server (first / last edge computing node) as model B.” (Palanisamy, page 3, right column, paragraph 2)
Said method comprising:
sampling input data samples from an input data set comprising a plurality of input data samples, and sampling ground truth data samples from a ground truth dataset comprising a plurality of ground truth data samples, each ground truth data sample in the ground truth dataset corresponding with one of the input data samples in the input dataset:
“First, to examine whether the proposed modification to backpropagation works, we simulate the new training procedure on the university server using VGG and DenseNet on CIFAR 100 [16] dataset.” (Palanisamy, page 4, right column, paragraph 4)
“Data can be input in batches. Let 𝐵 denote the batch size of the data. We input the data to model A.” (Palanisamy, page 3, right column, paragraph 3)
“Thus, we only exchange the output of intermediate layers that conveys no meta information about the input data or labels. For example, in the case of image classification, the images (input data samples) are input only to the mobile-side and then transformed to floating points (irrelevant to an outside entity) after the first few layers. Next, the output returned from the server is propagated through the last few layers on the mobile side before comparing that to the labels (ground truth data samples) stored only on the mobile side.” (Palanisamy, page 2, left column, paragraph 1)
“the output
y
^
of the network with input 𝑥 is:
y
^
=
f
C
(
f
B
f
A
x
)
(1)
Let
y
t
be the target (ground truth) for the network. The loss L calculated using the objective function J can be represented as:
L
=
J
(
y
t
,
y
^
)
” (Palanisamy, page 2, right column, paragraph 3)
transmitting, to a first edge computing node configured to perform computations of at least a first intermediate layer of at least one intermediate layer of a neural network, the input kernel matrix and the label kernel matrix:
PNG
media_image1.png
227
742
media_image1.png
Greyscale
”Overview of split learning with a simple NN “ (Palanisamy, page 3, right column, Figure 2)
“After propagating through the first few layers of the network, model A sends the generated output to Model B (first edge computing node) through a POST request … Model B propagates the received data through its layers (that is, most of the hidden layers (intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C listening on its end, receives the message and sends a GET request to model B to get its output values … These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3)
performing computations of an input layer of the neural network, on the basis of the input kernel matrix and the label kernel matrix and parameters of the input layer, to generate an input layer activation map:
“The forward pass equations for each of the layers can be represented as:
PNG
media_image2.png
91
486
media_image2.png
Greyscale
where {a, g(x), U,
U
b
}, {b, h(x), V,
V
b
}, {c, i(x), W,
W
b
}, {
y
^
, j(x), Z,
Z
b
} represent the outputs, activation functions, weights, and biases of each of the layers respectively (Palanisamy, page 2, right column, paragraph 4)
“as shown in Figure 1, for a multilayered network, we execute the first few layers on the device and then send the output for further training to a server … For example, in the case of image classification, the images are input only to the mobile-side and then transformed to floating points (irrelevant to an outside entity) after the first few layers (input layer[s])” (Palanisamy, page 1, right column, paragraph 3). The first (input) layers receive image input transformed into a set of activations as it’s passed forward.
updating the parameters of the input layer on the basis of a loss computed for the input layer based on an input activation kernel matrix and the label kernel matrix, the input activation kernel matrix determined based on the input layer activation map:
“the output
y
^
of the network with input 𝑥 is:
y
^
=
f
C
(
f
B
f
A
x
)
(1)
Let
y
t
be the target (ground truth label) for the network. The loss L calculated using the objective function J can be represented as:
L
=
J
(
y
t
,
y
^
)
” (Palanisamy, page 2, right column, paragraph 3)
PNG
media_image1.png
227
742
media_image1.png
Greyscale
”Overview of split learning with a simple NN “ (Palanisamy, page 3, right column, Figure 2)
“Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers. Note that the output from model B acts as the input layer for model C; we compute the gradients of this layer as well and then issue a POST request from model C to model B to send these gradients. Model B uses these gradients to generate its labels. Once the labels are obtained, we follow the standard procedure to compute the loss and update the model’s weights. Next, similar to model C, model B computes the gradient of its input layer (that is, the output received from model A) and notifies model A to collect it. The size of these gradients is equal to 𝐵 · 𝑅 · 𝐶 · 𝐹 . Model A (model containing the input layer[s]) then collects these gradients using a GET request and uses them to generate its labels. After that, we follow the standard procedure for backpropagation” (Palanisamy, page 3, right column, paragraph 4)
transmitting, to the first edge computing node, the input layer activation map:
“SplitEasy can function between any number of devices, but for this paper, we will stick with a basic client-server architecture as shown in Figure 2. The client is a mobile device that acts as both the starting and the ending point. That is, at any point of time, there are two separate instances running in parallel on the mobile device. We denote the start point as model A and the endpoint as model C. We use a single stand-alone server (first edge computing node) as model B.” (Palanisamy, page 3, right column, paragraph 2)
“After propagating through the first few layers of the network, model A sends the generated output to Model B (first edge computing node) through a POST request.” (Palanisamy, page 3, right column, paragraph 3)
receiving, from a last edge computing node configured to perform computations of a last intermediate layer of the at least one intermediate layer of the neural network, a last intermediate layer activation map: “Model B (last edge computing node) propagates the received data through its layers (that is, most of the hidden layers (intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C listening on its end, receives the message and sends a GET request to model B to get its output values (last intermediate layer activation map). The size of the data received is equal to 𝐵 · 𝐸, where 𝐸 denotes the embedding size. These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3)
performing computations of an output layer of the neural network, on the basis of the last intermediate layer activation map and parameters of the output layer, to generate an inference result comprising predicted labels:
PNG
media_image1.png
227
742
media_image1.png
Greyscale
(Palanisamy, page 3, right column, Figure 2)
“Next, the output (last intermediate layer activation map) returned from the server is propagated through the last few layers (output layer[s]) on the mobile side before comparing that (predicted labels) to the labels stored only on the mobile side.” (Palanisamy, page 2, left column, paragraph 1)
“Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers.” (Palanisamy, page 3, right column, paragraph 4)
updating the parameters of the output layer on the basis of a loss computed for the output layer on the basis the sampled training data labels and the predicted labels: “Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers (output layer[s])” (Palanisamy, page 3, right column, paragraph 4)
Palanisamy relates to split learning with edge nodes processing intermediate neural network layers and is analogous to the claimed invention.
While Palanisamy fails to disclose the further limitations of the claim, Ma discloses a method comprising:
sampling input data samples from an input data set comprising a plurality of input data samples, and sampling ground truth data samples from a ground truth dataset comprising a plurality of ground truth data samples, each ground truth data sample in the ground truth dataset corresponding with one of the input data samples in the input dataset:
“Implementing the Information Bottleneck principle, we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
where X ∈
R
m
×
d
x
is the input, Y ∈
R
m
×
d
y
is the label (ground truth), i ∈ {0, ...,L} and L is the number of hidden layers,
d
x
and
d
y
are the dimensionalities of the input and output variables respectively” (Ma, page 5087, right column, paragraph 3)
“We demonstrate that it is possible to train deep classification networks using an information bottleneck principle, without backpropagation, and obtain results competitive with the standard backpropagation optimization of the cross-entropy objective.” (Ma, page 5085, right column, page 4)
“For the experiments, we used standard feedforward networks with batch-normalization (Ioffe and Szegedy 2015) on the MNIST/Fashion MNIST/CIFAR10 datasets (input / ground truth dataset)” (Ma, page 5089, left column, paragraph 1)
determining an input kernel matrix based on the sampled input data samples; determining a label kernel matrix based on the sampled ground truth data samples:
“Due to the practical difficulties of calculating the mutual information among the random variables, we adopt a non-parametric kernel-based method, the Hilbert- Schmidt independence criterion (HSIC), to characterize the statistical (in)dependence of different layers. That is, for each network layer we maximize HSIC between the layer activation and the desired output (ground truth) and minimize HSIC between that layer activation and the input” (Ma, page 5085, right column, paragraph 2)
“Implementing the Information Bottleneck principle, we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
where X ∈
R
m
×
d
x
is the input, Y ∈
R
m
×
d
y
is the label (ground truth), i ∈ {0, ...,L} and L is the number of hidden layers,
d
x
and
d
y
are the dimensionalities of the input and output variables respectively … Following nHSIC of (5), the nHSIC of each term is:
PNG
media_image4.png
97
375
media_image4.png
Greyscale
” (Ma, page 5087, right column, paragraph 3)
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(input kernel matrix) and
K
-
Y
(label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2)
transmitting, to a first edge computing node configured to perform computations of at least a first intermediate layer of at least one intermediate layer of a neural network, the input kernel matrix and the label kernel matrix:
“for each network layer we maximize HSIC between the layer activation and the desired output and minimize HSIC between that layer activation and the input. In some cases, the final-layer representations resulting from this HSIC bottleneck training are nearly one-hot and can be directly used for classification after identifying a single fixed permutation.” (Ma, page 5085, right column, paragraph 2)
“we try using the HSIC-bottleneck as the sole training objective. We use a fully connected network architecture 784- 256-256-256-256-256-10 with ReLU activation functions” (Ma, page 5089, left column, paragraph 4). As discussed above, the HSIC-bottleneck objective requires an input kernel matrix and a label kernel matrix to be calculated.
PNG
media_image6.png
164
354
media_image6.png
Greyscale
”The HSIC-trained network (Fig. 1a) is a standard feedforward network trained using the HSIC-bottleneck objective, resulting in hidden representations at the last layer that can easily be used for classification” (Ma, page 5087, right column, Figure 1). All layers, including intermediate layers of a neural network, are trained.
performing computations of an input layer of the neural network, on the basis of the input kernel matrix and the label kernel matrix and parameters of the input layer, to generate an input layer activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2);
"
3
This can be extended to the activations produced from convolutional layers, as each activation is flattened and stacked in array
Z
i
” (Ma, page 5087, right column, footnote 3). For each layer to be trained, each layer must calculate its activation
Z
i
needed for the objective determination. As discussed above, the nHSIC calculations require kernel matrices for their two inputs. For the objective above, they thus require a kernel matrix for X (the input) and Y (the label).
PNG
media_image7.png
416
557
media_image7.png
Greyscale
(Ma, page 5088). As evident from line 10 of the above algorithm, each layer takes the previous layer’s activation (
Z
i
-
1
) as input for calculating its own activation (
Z
i
).
PNG
media_image6.png
164
354
media_image6.png
Greyscale
(Ma, page 5087, right column, Figure 1). All layers, including the input layer of a neural network, are trained, and all activations, including the activation of the input layer (
Z
0
), are calculated and propagated.
updating the parameters of the input layer on the basis of a loss computed for the input layer based on an input activation kernel matrix and the label kernel matrix, the input activation kernel matrix determined based on the input layer activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective (loss):
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2).
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(input kernel matrix) and
K
-
Y
(label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2). As demonstrated by these formulas, for some random variable R,
K
~
R
=
K
-
R
(
K
-
R
+
ϵ
m
I
m
)
-
1
, with
K
-
R
being a kernel matrix for R.
“Following nHSIC of (5), the nHSIC of each term is:
PNG
media_image4.png
97
375
media_image4.png
Greyscale
.” (Ma, page 5087, right column, paragraph 3). As per the formulas discussed above,
K
~
Z
i
=
K
-
Z
i
(
K
-
Z
i
+
ϵ
m
I
m
)
-
1
, with
K
-
Z
i
being a kernel matrix for
Z
i
(the activation).
transmitting, to the first edge computing node, the input layer activation map; receiving, from a last edge computing node configured to perform computations of a last intermediate layer of the at least one intermediate layer of the neural network, a last intermediate layer activation map
PNG
media_image6.png
164
354
media_image6.png
Greyscale
(Ma, page 5087, right column, Figure 1). All layers, including the input layer and intermediate layers of a neural network, are trained, and all activations, including the activation of the input layer (
Z
0
) and of the last intermediate layer (
Z
L
), are calculated and propagated.
performing computations of an output layer of the neural network, on the basis of the last intermediate layer activation map and parameters of the output layer, to generate an inference result comprising predicted labels:
“We append a single output layer O(·) :
R
d
L
→
R
d
y
equipped with a softmax function for classification (Fig. 1), taking the optimized last-layer hidden representation (last intermediate layer activation) from a unformatted-trained network as its input” (Ma, page 2088, left column, paragraph 5)
“We evaluate two specific approaches to produce classifications (predicted labels) from the HSIC-bottleneck trained network. First, if the outputs are one-hot, they can simply be permuted to align with the training labels” (Ma, page 5087, right column, paragraph 1)
updating the parameters of the output layer on the basis of a loss computed for the output layer on the basis the sampled training data labels and the predicted labels:
“We append a single output layer O(·) :
R
d
L
→
R
d
y
equipped with a softmax function for classification (Fig. 1), taking the optimized last-layer hidden representation from a unformatted-trained network as its input. The new layer is trained using minibatch SGD with the loss
PNG
media_image8.png
62
469
media_image8.png
Greyscale
where CE denotes the cross entropy loss” (Ma, page 2088, left column, paragraph 5). This loss compares the predicted labels (O(
Z
L
)) with the ground truth label (Y) (sampled training data labels).
Ma relates to using HSIC-modified information bottleneck loss to train a neural network and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 2, the rejection of claim 1 is incorporated. Ma further discloses a method, wherein updating the parameters of the input layer includes: computing an input layer gradient; and stepping the input layer gradient to update for the parameters of the input layer:
PNG
media_image7.png
416
557
media_image7.png
Greyscale
(Ma, page 5088). As seen in lines 10-12, the gradient is calculated and stepped for the input and each intermediate layer.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 3, the rejection of claim 1 is incorporated. Ma further discloses a method, wherein the loss for the input layer is computed using a modified information bottleneck (MIB) loss function:
“As is the case with cross entropy, an algorithm trained exclusively using mutual information is vulnerable to overfitting. To address this we train the network using an approximation of the information bottleneck (Tishby, Pereira, and Bialek 1999). Due to the practical difficulties of calculating the mutual information among the random variables, we adopt a non-parametric kernel-based method, the Hilbert- Schmidt independence criterion (HSIC), to characterize the statistical (in)dependence of different layers” (Ma, page 5085, right column, paragraph 2).
“The IB (information bottleneck) objective (loss) is
PNG
media_image9.png
67
552
media_image9.png
Greyscale
” (Ma, page 5086, left column, paragraph 5)
“In this paper, we replace the mutual information terms in the information bottleneck objective (loss) with HSIC” (Ma, page 5086, right column, paragraph 2)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 4, the rejection of claim 3 is incorporated. Ma further discloses a method, wherein the MIB loss function is based on a Hilbert-Schmidt Independence Criterion (HSIC): “In this paper, we replace the mutual information terms in the information bottleneck objective (loss) with HSIC” (Ma, page 5086, right column, paragraph 2)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 5, the rejection of claim 1 is incorporated. Ma further discloses a method, wherein updating the parameters of the output layer includes: computing an output layer gradient; and stepping the output layer gradient to update the parameters of the output layer: “We append a single output layer O(·) :
R
d
L
→
R
d
y
equipped with a softmax function for classification (Fig. 1), taking the optimized last-layer hidden representation from a unformatted-trained network as its input. The new layer is trained using minibatch SGD (stochastic gradient descent) with the loss
PNG
media_image8.png
62
469
media_image8.png
Greyscale
where CE denotes the cross entropy loss” (Ma, page 2088, left column, paragraph 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 7, the rejection of claim 5 is incorporated. Ma further discloses a method, wherein the loss function is a cross-entropy loss function: “We append a single output layer O(·) :
R
d
L
→
R
d
y
equipped with a softmax function for classification (Fig. 1), taking the optimized last-layer hidden representation from a unformatted-trained network as its input. The new layer is trained using minibatch SGD with the loss
PNG
media_image8.png
62
469
media_image8.png
Greyscale
where CE denotes the cross entropy loss” (Ma, page 2088, left column, paragraph 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Palanisamy (SplitEasy: A Practical Approach for Training ML models on Mobile Devices, published 1/29/2021, arXiv:2011.04232v2) in view of Ma (The HSIC Bottleneck: Deep Learning without Back-Propagation, published 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) pp. 5085-5092), and further in view of Murphy (FEATURE EXTRACTION AND MACHINE LEARNING FOR AUTOMATED METADATA ANALYSIS, published 10/24/2019, US 2019/0325259 A1).
Regarding claim 6, the rejection of claim 5 is incorporated. While the aforementioned references fail to disclose the further limitations of the claim, Sundaram discloses a method, wherein the loss of the output layer is computed using a mean squared error loss function: “The multi-label classification algorithm includes comparing the multi-label classification output of a discriminatively categorized object detection label set associated with an image from the training set to the ground truth label set for that same image. Comparing can include measuring a loss between the multi-label classification output of a discriminatively categorized object detection label set and the ground truth label set using maximum likelihood estimation, mean squared error, Hamming distance, and/or Jaccard difference” (Murphy, [0018]).
Murphy relates to using mean-squared error as a loss term comparing neural network classifier outputs and ground truth targets and is analogous to the claimed invention. The existing combination teaches a method of training the output layer of a neural network using cross-entropy loss. The claimed invention differs from this method by using a mean squared error loss function. Murphy teaches a mean squared error loss function for training network outputs. Because both the existing combination and Murphy teach the use of training network classifier outputs with loss functions, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to substitute the existing combination’s cross-entropy for mean-squared error to achieve the predictable result of a loss function that penalizes network outputs more the further they are from the ground truth target data (MPEP 2143 I. (B) Substituting one known element for another for predictable results).
Claims 8 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Palanisamy (SplitEasy: A Practical Approach for Training ML models on Mobile Devices, published 1/29/2021, arXiv:2011.04232v2) in view of Ma (The HSIC Bottleneck: Deep Learning without Back-Propagation, published 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) pp. 5085-5092), and further in view of Chen (TRAINING GIANT NEURAL NETWORKS USING PIPELINE PARALLELISM, published 2/11/2021, US 20210042620 A1).
Regarding claim 8, the rejection of claim 1 is incorporated. Ma further discloses a method, wherein the first edge computing node and the last edge computing node are among a plurality of edge computing nodes, and transmitting, to the first edge computing node configured to perform computations of at least the first intermediate layer of the at least one intermediate layer of the neural network, the input kernel matrix and the label kernel matrix comprises: broadcasting, to the plurality of edge computing nodes, the input kernel matrix and the label kernel matrix:
PNG
media_image7.png
416
557
media_image7.png
Greyscale
(Ma, page 5088). As seen in lines 8-14 of the algorithm above, the nHSIC objective is calculated over all data for each layer. As discussed regarding claim 1, this calculation depends on kernel matrices for the input and the labels, which are constant across different layers. Thus, this information is being broadcast to each layer.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
While the aforementioned references fail to disclose the further limitations of the claim, Chen discloses a method, wherein the first edge computing node and the last edge computing node are among a plurality of computing nodes:
“Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of network parameters.” (Chen, [0003])
“The operations include: obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices (plurality of edge computing nodes) … Each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network” (Chen, [0008])
“The described system allows for effectively distributing the training of giant neural networks, i.e., neural networks with a very large number of parameters, across multiple computing devices. By partitioning the neural network into a plurality of composite layers, the described system can scale arbitrary giant neural network architectures beyond the memory limitations of a single computing device” (Chen, [0013])
“The system 100 can receive the training examples 101 and the data specifying the giant neural network 102 in any of a variety of ways. For example, the system 100 can receive training data as an upload from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system 100” (Chen, [0029])
Chen relates to split learning across distributed systems for neural networks and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to distribute the server’s network across a plurality of devices using the method disclosed by Chen. Doing so would enable arbitrarily large neural networks to be trained by exceeding the memory limitations of a single computing device. See Chen, [0013].
Regarding claim 15, Palanisamy discloses a local computing node … to carry out operations: “SplitEasy can function between any number of devices, but for this paper, we will stick with a basic client-server architecture as shown in Figure 2. The client (local computing node) is a mobile device that acts as both the starting and the ending point. That is, at any point of time, there are two separate instances running in parallel on the mobile device. We denote the start point as model A and the endpoint as model C. We use a single stand-alone server (first / last edge computing node) as model B.” (Palanisamy, page 3, right column, paragraph 2)
Said operations comprising:
sampling input data samples from an input data set comprising a plurality of input data samples, and sampling ground truth data samples from a ground truth dataset comprising a plurality of ground truth data samples, each ground truth data sample in the ground truth dataset corresponding with one of the input data samples in the input dataset:
“First, to examine whether the proposed modification to backpropagation works, we simulate the new training procedure on the university server using VGG and DenseNet on CIFAR 100 [16] dataset.” (Palanisamy, page 4, right column, paragraph 4)
“Data can be input in batches. Let 𝐵 denote the batch size of the data. We input the data to model A.” (Palanisamy, page 3, right column, paragraph 3)
“Thus, we only exchange the output of intermediate layers that conveys no meta information about the input data or labels. For example, in the case of image classification, the images (input data samples) are input only to the mobile-side and then transformed to floating points (irrelevant to an outside entity) after the first few layers. Next, the output returned from the server is propagated through the last few layers on the mobile side before comparing that to the labels (ground truth data samples) stored only on the mobile side.” (Palanisamy, page 2, left column, paragraph 1)
“the output
y
^
of the network with input 𝑥 is:
y
^
=
f
C
(
f
B
f
A
x
)
(1)
Let
y
t
be the target (ground truth) for the network. The loss L calculated using the objective function J can be represented as:
L
=
J
(
y
t
,
y
^
)
” (Palanisamy, page 2, right column, paragraph 3)
transmitting, to a first edge computing node configured to perform computations of at least a first intermediate layer of at least one intermediate layer of a neural network, the input kernel matrix and the label kernel matrix:
PNG
media_image1.png
227
742
media_image1.png
Greyscale
”Overview of split learning with a simple NN “ (Palanisamy, page 3, right column, Figure 2)
“After propagating through the first few layers of the network, model A sends the generated output to Model B (first edge computing node) through a POST request … Model B propagates the received data through its layers (that is, most of the hidden layers (intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C listening on its end, receives the message and sends a GET request to model B to get its output values … These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3)
performing computations of an input layer of the neural network, on the basis of the input kernel matrix and the label kernel matrix and parameters of the input layer, to generate an input layer activation map:
“The forward pass equations for each of the layers can be represented as:
PNG
media_image2.png
91
486
media_image2.png
Greyscale
where {a, g(x), U,
U
b
}, {b, h(x), V,
V
b
}, {c, i(x), W,
W
b
}, {
y
^
, j(x), Z,
Z
b
} represent the outputs, activation functions, weights, and biases of each of the layers respectively (Palanisamy, page 2, right column, paragraph 4)
“as shown in Figure 1, for a multilayered network, we execute the first few layers on the device and then send the output for further training to a server … For example, in the case of image classification, the images are input only to the mobile-side and then transformed to floating points (irrelevant to an outside entity) after the first few layers (input layer[s])” (Palanisamy, page 1, right column, paragraph 3). The first (input) layers receive image input transformed into a set of activations as it’s passed forward.
updating the parameters of the input layer on the basis of a loss computed for the input layer based on an input activation kernel matrix and the label kernel matrix, the input activation kernel matrix determined based on the input layer activation map:
“the output
y
^
of the network with input 𝑥 is:
y
^
=
f
C
(
f
B
f
A
x
)
(1)
Let
y
t
be the target (ground truth label) for the network. The loss L calculated using the objective function J can be represented as:
L
=
J
(
y
t
,
y
^
)
” (Palanisamy, page 2, right column, paragraph 3)
PNG
media_image1.png
227
742
media_image1.png
Greyscale
”Overview of split learning with a simple NN “ (Palanisamy, page 3, right column, Figure 2)
“Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers. Note that the output from model B acts as the input layer for model C; we compute the gradients of this layer as well and then issue a POST request from model C to model B to send these gradients. Model B uses these gradients to generate its labels. Once the labels are obtained, we follow the standard procedure to compute the loss and update the model’s weights. Next, similar to model C, model B computes the gradient of its input layer (that is, the output received from model A) and notifies model A to collect it. The size of these gradients is equal to 𝐵 · 𝑅 · 𝐶 · 𝐹 . Model A (model containing the input layer[s]) then collects these gradients using a GET request and uses them to generate its labels. After that, we follow the standard procedure for backpropagation” (Palanisamy, page 3, right column, paragraph 4)
transmitting, to the first edge computing node, the input layer activation map:
“SplitEasy can function between any number of devices, but for this paper, we will stick with a basic client-server architecture as shown in Figure 2. The client is a mobile device that acts as both the starting and the ending point. That is, at any point of time, there are two separate instances running in parallel on the mobile device. We denote the start point as model A and the endpoint as model C. We use a single stand-alone server (first edge computing node) as model B.” (Palanisamy, page 3, right column, paragraph 2)
“After propagating through the first few layers of the network, model A sends the generated output to Model B (first edge computing node) through a POST request.” (Palanisamy, page 3, right column, paragraph 3)
receiving, from a last edge computing node configured to perform computations of a last intermediate layer of the at least one intermediate layer of the neural network, a last intermediate layer activation map: “Model B (last edge computing node) propagates the received data through its layers (that is, most of the hidden layers (intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C listening on its end, receives the message and sends a GET request to model B to get its output values (last intermediate layer activation map). The size of the data received is equal to 𝐵 · 𝐸, where 𝐸 denotes the embedding size. These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3)
performing computations of an output layer of the neural network, on the basis of the last intermediate layer activation map and parameters of the output layer, to generate an inference result comprising predicted labels:
PNG
media_image1.png
227
742
media_image1.png
Greyscale
(Palanisamy, page 3, right column, Figure 2)
“Next, the output (last intermediate layer activation map) returned from the server is propagated through the last few layers (output layer[s]) on the mobile side before comparing that (predicted labels) to the labels stored only on the mobile side.” (Palanisamy, page 2, left column, paragraph 1)
“Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers.” (Palanisamy, page 3, right column, paragraph 4)
updating the parameters of the output layer on the basis of a loss computed for the output layer on the basis the sampled training data labels and the predicted labels: “Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers (output layer[s])” (Palanisamy, page 3, right column, paragraph 4)
Palanisamy relates to split learning with edge nodes processing intermediate neural network layers and is analogous to the claimed invention.
While Palanisamy fails to disclose the further limitations of the claim, Ma discloses a operations comprising:
sampling input data samples from an input data set comprising a plurality of input data samples, and sampling ground truth data samples from a ground truth dataset comprising a plurality of ground truth data samples, each ground truth data sample in the ground truth dataset corresponding with one of the input data samples in the input dataset:
“Implementing the Information Bottleneck principle, we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
where X ∈
R
m
×
d
x
is the input, Y ∈
R
m
×
d
y
is the label (ground truth), i ∈ {0, ...,L} and L is the number of hidden layers,
d
x
and
d
y
are the dimensionalities of the input and output variables respectively” (Ma, page 5087, right column, paragraph 3)
“We demonstrate that it is possible to train deep classification networks using an information bottleneck principle, without backpropagation, and obtain results competitive with the standard backpropagation optimization of the cross-entropy objective.” (Ma, page 5085, right column, page 4)
“For the experiments, we used standard feedforward networks with batch-normalization (Ioffe and Szegedy 2015) on the MNIST/Fashion MNIST/CIFAR10 datasets (input / ground truth dataset)” (Ma, page 5089, left column, paragraph 1)
determining an input kernel matrix based on the sampled input data samples; determining a label kernel matrix based on the sampled ground truth data samples:
“Due to the practical difficulties of calculating the mutual information among the random variables, we adopt a non-parametric kernel-based method, the Hilbert- Schmidt independence criterion (HSIC), to characterize the statistical (in)dependence of different layers. That is, for each network layer we maximize HSIC between the layer activation and the desired output (ground truth) and minimize HSIC between that layer activation and the input” (Ma, page 5085, right column, paragraph 2)
“Implementing the Information Bottleneck principle, we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
where X ∈
R
m
×
d
x
is the input, Y ∈
R
m
×
d
y
is the label (ground truth), i ∈ {0, ...,L} and L is the number of hidden layers,
d
x
and
d
y
are the dimensionalities of the input and output variables respectively … Following nHSIC of (5), the nHSIC of each term is:
PNG
media_image4.png
97
375
media_image4.png
Greyscale
” (Ma, page 5087, right column, paragraph 3)
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(input kernel matrix) and
K
-
Y
(label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2)
transmitting, to a first edge computing node configured to perform computations of at least a first intermediate layer of at least one intermediate layer of a neural network, the input kernel matrix and the label kernel matrix:
“for each network layer we maximize HSIC between the layer activation and the desired output and minimize HSIC between that layer activation and the input. In some cases, the final-layer representations resulting from this HSIC bottleneck training are nearly one-hot and can be directly used for classification after identifying a single fixed permutation.” (Ma, page 5085, right column, paragraph 2)
“we try using the HSIC-bottleneck as the sole training objective. We use a fully connected network architecture 784- 256-256-256-256-256-10 with ReLU activation functions” (Ma, page 5089, left column, paragraph 4). As discussed above, the HSIC-bottleneck objective requires an input kernel matrix and a label kernel matrix to be calculated.
PNG
media_image6.png
164
354
media_image6.png
Greyscale
”The HSIC-trained network (Fig. 1a) is a standard feedforward network trained using the HSIC-bottleneck objective, resulting in hidden representations at the last layer that can easily be used for classification” (Ma, page 5087, right column, Figure 1). All layers, including intermediate layers of a neural network, are trained.
performing computations of an input layer of the neural network, on the basis of the input kernel matrix and the label kernel matrix and parameters of the input layer, to generate an input layer activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2);
"
3
This can be extended to the activations produced from convolutional layers, as each activation is flattened and stacked in array
Z
i
” (Ma, page 5087, right column, footnote 3). For each layer to be trained, each layer must calculate its activation
Z
i
needed for the objective determination. As discussed above, the nHSIC calculations require kernel matrices for their two inputs. For the objective above, they thus require a kernel matrix for X (the input) and Y (the label).
PNG
media_image7.png
416
557
media_image7.png
Greyscale
(Ma, page 5088). As evident from line 10 of the above algorithm, each layer takes the previous layer’s activation (
Z
i
-
1
) as input for calculating its own activation (
Z
i
).
PNG
media_image6.png
164
354
media_image6.png
Greyscale
(Ma, page 5087, right column, Figure 1). All layers, including the input layer of a neural network, are trained, and all activations, including the activation of the input layer (
Z
0
), are calculated and propagated.
updating the parameters of the input layer on the basis of a loss computed for the input layer based on an input activation kernel matrix and the label kernel matrix, the input activation kernel matrix determined based on the input layer activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective (loss):
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2).
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(input kernel matrix) and
K
-
Y
(label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2). As demonstrated by these formulas, for some random variable R,
K
~
R
=
K
-
R
(
K
-
R
+
ϵ
m
I
m
)
-
1
, with
K
-
R
being a kernel matrix for R.
“Following nHSIC of (5), the nHSIC of each term is:
PNG
media_image4.png
97
375
media_image4.png
Greyscale
.” (Ma, page 5087, right column, paragraph 3). As per the formulas discussed above,
K
~
Z
i
=
K
-
Z
i
(
K
-
Z
i
+
ϵ
m
I
m
)
-
1
, with
K
-
Z
i
being a kernel matrix for
Z
i
(the activation).
transmitting, to the first edge computing node, the input layer activation map; receiving, from a last edge computing node configured to perform computations of a last intermediate layer of the at least one intermediate layer of the neural network, a last intermediate layer activation map
PNG
media_image6.png
164
354
media_image6.png
Greyscale
(Ma, page 5087, right column, Figure 1). All layers, including the input layer and intermediate layers of a neural network, are trained, and all activations, including the activation of the input layer (
Z
0
) and of the last intermediate layer (
Z
L
), are calculated and propagated.
performing computations of an output layer of the neural network, on the basis of the last intermediate layer activation map and parameters of the output layer, to generate an inference result comprising predicted labels:
“We append a single output layer O(·) :
R
d
L
→
R
d
y
equipped with a softmax function for classification (Fig. 1), taking the optimized last-layer hidden representation (last intermediate layer activation) from a unformatted-trained network as its input” (Ma, page 2088, left column, paragraph 5)
“We evaluate two specific approaches to produce classifications (predicted labels) from the HSIC-bottleneck trained network. First, if the outputs are one-hot, they can simply be permuted to align with the training labels” (Ma, page 5087, right column, paragraph 1)
updating the parameters of the output layer on the basis of a loss computed for the output layer on the basis the sampled training data labels and the predicted labels:
“We append a single output layer O(·) :
R
d
L
→
R
d
y
equipped with a softmax function for classification (Fig. 1), taking the optimized last-layer hidden representation from a unformatted-trained network as its input. The new layer is trained using minibatch SGD with the loss
PNG
media_image8.png
62
469
media_image8.png
Greyscale
where CE denotes the cross entropy loss” (Ma, page 2088, left column, paragraph 5). This loss compares the predicted labels (O(
Z
L
)) with the ground truth label (Y) (sampled training data labels).
Ma relates to using HSIC-modified information bottleneck loss to train a neural network and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
While Ma fails to disclose the further limitations of the claim, Chen discloses one or more processors; and a memory storing instructions which, when executed by the one or more processors
cause the one or more processors to carry out operations: “Each device 106 can have access to a memory (120) in the system 100, e.g., a random access memory ("RAM"), for storing instructions and data and data, e.g., a matrix or matrices representing network parameters for all network layers of a composite layer and instructions for executing forward computation functions and backpropagation functions. … The devices can include accelerators such as GPUs and CPUs (processors), as well as specialized hardware resources for efficiently performing certain operations, e.g., matrix multiplication, used in training a neural network” (Chen, [0048])
Chen relates to split learning across distributed systems for neural networks and is analogous to the claimed invention. The existing combination teaches a distributed neural network training system. The claimed invention improves upon this method by storing it in the form of instructions on computer hardware. Chen teaches computer hardware storing instructions to execute a distributed neural network training system, applicable to the existing combination. A person of ordinary skill in the art would have recognized that storing the existing combination’s method as computer instructions on Chen’s hardware would lead to the predictable result of the method being executable by a computing system, and would improve the known device by allowing it to be performed with real data (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
The analysis of claims 16-19 mirrors that of claims 2-5, with the exception that claims 16-19 are directed to generic computer hardware which executes the methods of claims 2-5. This generic hardware is taught by Chen, as discussed regarding claim 15. Thus, claims 16-19 are rejected under the same rationales used for claims 2-5, respectively.
Regarding claim 20, Palanisamy discloses [a]n edge computing node: “SplitEasy can function between any number of devices, but for this paper, we will stick with a basic client-server architecture as shown in Figure 2. The client (local computing node) is a mobile device that acts as both the starting and the ending point. That is, at any point of time, there are two separate instances running in parallel on the mobile device. We denote the start point as model A and the endpoint as model C. We use a single stand-alone server (edge computing node) as model B.” (Palanisamy, page 3, right column, paragraph 2)
Said edge computing node performing operations comprising:
performing computations of a first intermediate layer of at least one intermediate layer of a neural network:
PNG
media_image1.png
227
742
media_image1.png
Greyscale
”Overview of split learning with a simple NN “ (Palanisamy, page 3, right column, Figure 2)
“After propagating through the first few layers of the network, model A sends the generated output to Model B (edge computing node) through a POST request … Model B propagates the received data through its layers (that is, most of the hidden layers (intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C listening on its end, receives the message and sends a GET request to model B to get its output values … These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3)
receiving a training request, where the training request is specific to the edge computing node and includes an indication of an input edge computing node and an output edge computing node; receiving, for a local computing node, an input kernel matrix and an label kernel matrix: “Since model C (output computing node) is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers. Note that the output from model B acts as the input layer for model C; we compute the gradients of this layer as well and then issue a POST request from model C to model B to send these gradients. Model B uses these gradients to generate its labels. Once the labels are obtained, we follow the standard procedure to compute the loss and update the model’s weights. Next, similar to model C, model B computes the gradient of its input layer (that is, the output received from model A) and notifies model A (local / input computing node) to collect it” (Palanisamy, page 3, right column, paragraph 4)
receiving, from the input computing node, an input activation map:
“The forward pass equations for each of the layers can be represented as:
PNG
media_image2.png
91
486
media_image2.png
Greyscale
where {a, g(x), U,
U
b
}, {b, h(x), V,
V
b
}, {c, i(x), W,
W
b
}, {
y
^
, j(x), Z,
Z
b
} represent the outputs, activation functions, weights, and biases of each of the layers respectively (Palanisamy, page 2, right column, paragraph 4)
“as shown in Figure 1, for a multilayered network, we execute the first few layers on the device (input computing node) and then send the output (input activation map) for further training to a server … For example, in the case of image classification, the images are input only to the mobile-side and then transformed to floating points (irrelevant to an outside entity) after the first few layers” (Palanisamy, page 1, right column, paragraph 3).
performing computations of the first intermediate layer, on the basis of the input kernel matrix and the label kernel matrix and parameters of the input first intermediate layer, to generate an intermediate layer activation map: “After propagating through the first few layers of the network, model A sends the generated output to Model B through a POST request … Model B propagates the received data through its layers (that is, most of the hidden layers (layers including intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C listening on its end, receives the message and sends a GET request to model B to get its output values (intermediate layer activation map) … These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3). As noted in the rejections under 35 U.S.C. 112(b), “the input first intermediate layer” is interpreted as “the first intermediate layer”.
updating the parameters of the first intermediate layer on the basis of a loss computed for the first intermediate layer based on an input activation kernel matrix and the label kernel matrix, the input activation kernel matrix determined based on the input activation map:
“the output
y
^
of the network with input 𝑥 is:
y
^
=
f
C
(
f
B
f
A
x
)
(1)
Let
y
t
be the target (ground truth label) for the network. The loss L calculated using the objective function J can be represented as:
L
=
J
(
y
t
,
y
^
)
” (Palanisamy, page 2, right column, paragraph 3)
“Since model C is the endpoint of the network, it holds all the labels for computing the loss of the network. For model C, we simply use these labels to compute the loss and gradients to backpropagate through its layers. Note that the output from model B acts as the input layer for model C; we compute the gradients of this layer as well and then issue a POST request from model C to model B to send these gradients. Model B (model holding the intermediate layer[s]) uses these gradients to generate its labels. Once the labels are obtained, we follow the standard procedure to compute the loss and update the model’s weights. Next, similar to model C, model B computes the gradient of its input layer (that is, the output received from model A) and notifies model A to collect it. The size of these gradients is equal to 𝐵 · 𝑅 · 𝐶 · 𝐹 . Model A (model containing the input layer[s]) then collects these gradients using a GET request and uses them to generate its labels. After that, we follow the standard procedure for backpropagation” (Palanisamy, page 3, right column, paragraph 4)
transmitting, to the output computing node, the first intermediate layer: “Model B propagates the received data through its layers (that is, most of the hidden layers (intermediate layer[s])) and broadcasts a message notifying the listeners of the task completion. Model C (output computing node) listening on its end, receives the message and sends a GET request to model B to get its output values (final activation calculated using the first intermediate layer’s activation) … These values are subsequently propagated through the layers of Model C.” (Palanisamy, page 3, right column, paragraph 3)
Palanisamy relates to split learning with edge nodes processing intermediate neural network layers and is analogous to the claimed invention.
While Palanisamy fails to disclose the further limitations of the claim, Ma discloses operations comprising:
performing computations of a first intermediate layer of at least one intermediate layer of a neural network:
“for each network layer we maximize HSIC between the layer activation and the desired output and minimize HSIC between that layer activation and the input.” (Ma, page 5085, right column, paragraph 2)
PNG
media_image6.png
164
354
media_image6.png
Greyscale
”The HSIC-trained network (Fig. 1a) is a standard feedforward network trained using the HSIC-bottleneck objective, resulting in hidden representations at the last layer that can easily be used for classification” (Ma, page 5087, right column, Figure 1). All layers, including intermediate layers of a neural network, are trained.
receiving, for a local computing node, an input kernel matrix and an label kernel matrix:
“Due to the practical difficulties of calculating the mutual information among the random variables, we adopt a non-parametric kernel-based method, the Hilbert- Schmidt independence criterion (HSIC), to characterize the statistical (in)dependence of different layers. That is, for each network layer we maximize HSIC between the layer activation and the desired output (ground truth) and minimize HSIC between that layer activation and the input” (Ma, page 5085, right column, paragraph 2)
“Implementing the Information Bottleneck principle, we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
where X ∈
R
m
×
d
x
is the input, Y ∈
R
m
×
d
y
is the label (ground truth), i ∈ {0, ...,L} and L is the number of hidden layers,
d
x
and
d
y
are the dimensionalities of the input and output variables respectively … Following nHSIC of (5), the nHSIC of each term is:
PNG
media_image4.png
97
375
media_image4.png
Greyscale
” (Ma, page 5087, right column, paragraph 3)
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(input kernel matrix) and
K
-
Y
(label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2)
receiving, from the input computing node, an input activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2);
"
3
This can be extended to the activations produced from convolutional layers, as each activation is flattened and stacked in array
Z
i
” (Ma, page 5087, right column, footnote 3). For each layer to be trained, each layer must calculate its activation
Z
i
needed for the objective determination.
PNG
media_image7.png
416
557
media_image7.png
Greyscale
(Ma, page 5088). As evident from line 10 of the above algorithm, each layer takes the previous layer’s activation (
Z
i
-
1
) as input for calculating its own activation (
Z
i
).
PNG
media_image6.png
164
354
media_image6.png
Greyscale
(Ma, page 5087, right column, Figure 1). All layers, including the input layer of a neural network, are trained, and all activations, including the activation of the input layer (
Z
0
), are calculated and propagated.
performing computations of the first intermediate layer, on the basis of the input kernel matrix and the label kernel matrix and parameters of the input first intermediate layer, to generate an intermediate layer activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective:
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2);
"
3
This can be extended to the activations produced from convolutional layers, as each activation is flattened and stacked in array
Z
i
” (Ma, page 5087, right column, footnote 3). For each layer to be trained, each layer must calculate its activation
Z
i
needed for the objective determination. As discussed above, the nHSIC calculations require kernel matrices for their two inputs. For the objective above, they thus require a kernel matrix for X (the input) and Y (the label).
PNG
media_image7.png
416
557
media_image7.png
Greyscale
(Ma, page 5088). As evident from line 10 of the above algorithm, each layer takes the previous layer’s activation (
Z
i
-
1
) as input for calculating its own activation (
Z
i
).
PNG
media_image6.png
164
354
media_image6.png
Greyscale
(Ma, page 5087, right column, Figure 1). All layers, including the intermediate layer[s] of a neural network, are trained, and all activations, including the activation of the first intermediate layer (
T
1
), are calculated and propagated.
updating the parameters of the first intermediate layer on the basis of a loss computed for the first intermediate layer based on an input activation kernel matrix and the label kernel matrix, the input activation kernel matrix determined based on the input activation map:
“Suppose we have a network composed of L hidden layers
T
i
∙
:
R
d
i
-
1
→
R
d
i
, resulting in hidden representation
s
3
… we replace the original mutual information terms with nHSIC (5) as the learning objective (loss):
PNG
media_image3.png
59
621
media_image3.png
Greyscale
” (Ma, page 5087, right column, paragraph 2).
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(input kernel matrix) and
K
-
Y
(label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2). As demonstrated by these formulas, for some random variable R,
K
~
R
=
K
-
R
(
K
-
R
+
ϵ
m
I
m
)
-
1
, with
K
-
R
being a kernel matrix for R.
“Following nHSIC of (5), the nHSIC of each term is:
PNG
media_image4.png
97
375
media_image4.png
Greyscale
.” (Ma, page 5087, right column, paragraph 3). As per the formulas discussed above,
K
~
Z
i
=
K
-
Z
i
(
K
-
Z
i
+
ϵ
m
I
m
)
-
1
, with
K
-
Z
i
being a kernel matrix for
Z
i
(the activation).
Ma relates to using HSIC-modified information bottleneck loss to train a neural network and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
While Ma fails to disclose the further limitations of the claim, Chen discloses [a]n edge computing node comprising one or more processors; and a memory storing instructions which, when executed by the one or more processors cause the one or more processors to carry out operations: “Each device 106 (edge computing node) can have access to a memory (120) in the system 100, e.g., a random access memory ("RAM"), for storing instructions and data and data, e.g., a matrix or matrices representing network parameters for all network layers of a composite layer and instructions for executing forward computation functions and backpropagation functions. … The devices can include accelerators such as GPUs and CPUs (processors), as well as specialized hardware resources for efficiently performing certain operations, e.g., matrix multiplication, used in training a neural network” (Chen, [0048])
Said operations comprising: receiving a training request, where the training request is specific to the edge computing node and includes an indication of an input edge computing node and an output edge computing node; receiving, for a local computing node, an input kernel matrix and an label kernel matrix:
“Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of network parameters.” (Chen, [0003])
“The operations include: obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices (plurality of edge computing nodes) … Each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network” (Chen, [0008])
“The described system allows for effectively distributing the training of giant neural networks, i.e., neural networks with a very large number of parameters, across multiple computing devices. By partitioning the neural network into a plurality of composite layers, the described system can scale arbitrary giant neural network architectures beyond the memory limitations of a single computing device” (Chen, [0013])
“The system 100 can receive the training examples 101 and the data specifying the giant neural network 102 in any of a variety of ways. For example, the system 100 can receive training data as an upload from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system 100” (Chen, [0029])
“the input for the succeeding computing device (output edge computing node) to perform the second part of the operations is the output from the preceding computing (input edge computing node) device performing the first part of the operations” (Chen, [0024])
Chen relates to split learning across distributed systems for neural networks and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to further distribute the existing combination’s network training system across a plurality of edge devices using the method disclosed by Chen. Doing so would enable arbitrarily large neural networks to be trained by exceeding the memory limitations of a single computing device. See Chen, [0013].
Claims 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over Palanisamy (SplitEasy: A Practical Approach for Training ML models on Mobile Devices, published 1/29/2021, arXiv:2011.04232v2) in view of Ma (The HSIC Bottleneck: Deep Learning without Back-Propagation, published 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) pp. 5085-5092), and further in view of Iwahashi (METHOD FOR PROCESSING SPEECH SIGNAL USING SUB-CONVERTING FUNCTIONS AND A WEIGHTING FUNCTION TO PRODUCE SYNTHESIZED SPEECH, patented 12/30/1997, Patent No. 5,704,006).
Regarding claim 9, the rejection of claim 1 is incorporated. Ma further discloses a method, wherein determining the input kernel matrix comprises defining the input kernel matrix as
K
A
i
j
=
k
-
(
a
i
,
a
j
)
, where
a
i
,
a
j
, are the sampled input data samples and
k
-
function represents a normalized Gaussian kernel function:
“Let
D
≔
{
x
1
,
y
1
,
…
,
x
m
,
y
m
}
contain m i.i.d. samples drawn from
P
X
Y
, where
x
i
∈
R
d
x
and
y
i
∈
R
d
y
. Then (2) leads to the following empirical expression (Gretton et al. 2005):
PNG
media_image10.png
38
579
media_image10.png
Greyscale
where
K
X
∈
R
m
×
m
(kernel matrix before centering) and
K
Y
∈
R
m
×
m
(kernel matrix before centering) have entries
K
X
i
j
=
k
(
x
i
,
x
j
)
(sampled input data) and
K
Y
i
j
=
k
(
y
i
,
y
j
)
, and
H
∈
R
m
x
m
is the centering matrix
H
=
I
m
-
1
m
1
m
1
m
T
” (Ma, page 5086, right column, paragraph 3)
“With an appropriate kernel choice such as the Gaussian k(x, y) … HSIC is zero if and only if the random variables X and Y are independent” (Ma, page 5086, right column, paragraph 4)
“In our experiments we use the normalized-HSIC (nHSIC) formulation based on the normalized cross-covariance operator (Fukumizu et al. 2008; Blaschko and Gretton 2008), given by
PNG
media_image5.png
43
399
media_image5.png
Greyscale
where
K
~
X
=
K
-
X
(
K
-
X
+
ϵ
m
I
m
)
-
1
and
K
~
Y
=
K
-
Y
(
K
-
Y
+
ϵ
m
I
m
)
-
1
.
K
-
X
(centered input kernel matrix) and
K
-
Y
(centered label kernel matrix) denote centered kernel matrices and
ϵ
is a small constant” (Ma, page 5087, left column, paragraph 2)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
While the aforementioned references fail to disclose the further limitations of the claim, Iwahashi discloses a method, wherein … the input data samples are one-dimensional vectors: “The radial basis function receives a one- or more dimensional vector as an input, and outputs a scalar value” (Iwahashi, column 4, paragraph 4). As noted in paragraph [0106] of the instant specification, “Radial Basis Function” is another name that can be used for the claimed “Gaussian kernel function”.
Iwahashi relates to Gaussian kernels receiving one-dimensional inputs and is analogous to the claimed invention. The existing combination teaches a method of applying Gaussian kernels to sampled input data of a neural network. The claimed invention differs from this method by applying Gaussian kernels to one-dimensional input data. Iwahashi teaches a method of applying Gaussian kernels to one-dimensional vector input data. Because both the existing combination and Iwahashi teach the use of Gaussian kernels applied to input data, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to substitute the existing combination’s input data of indiscriminate dimensionality for one-dimensional vectors to achieve the predictable result of a HSIC bottleneck-based neural network system applicable to one-dimensional vector input data (MPEP 2143 I. (B) Substituting one known element for another for predictable results).
Regarding claim 10, the rejection of claim 9 is incorporated. Ma further discloses a method, wherein computing the input kernel matrix comprises optimizing the normalized Gaussian kernel function
k
-
based on a tunable hyperparameter for establishing an extent to which the two one-dimensional vectors interact with each other when using the input kernel matrix to approximate mutual information: “While in principle HSIC can discover arbitrary dependencies between variables, in practice and with finite data the choice of the σ parameter (hyperparameter) in the HSIC kernel emphasizes relationships at some scales more than others. Intuitively, two data points x, y are not well distinguished when their difference is sufficiently small or large, such that they lie on the small-slope portions of the Gaussian. This is typically handled by choosing the kernel σ based on median distances among the data (Sejdinovic et al. 2012; Sugiyama and Yamada 2012), or by a hyperparameter search.” (Ma, page 5087, left column, paragraph 5). As discussed regarding claim 9 above, Iwahashi discloses using one-dimensional vectors as kernel input.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 11, the rejection of claim 10 is incorporated. Ma discloses a method, further comprising, before determining the normalized Gaussian kernel, selecting the tunable hyperparameter: “In our case, the data “points” are minibatches of activations from different network layers, and thus have different dimensionality. This suggests that different σ (hyperparameter) values should be used in each kernel. Using the observation that the expected squared distance between random points scales with dimension, these multiple hyperparameters can be approximated by scaling a single σ with the dimensionality d of points x and y as:
PNG
media_image11.png
63
688
media_image11.png
Greyscale
(used in all our experiments).” (Ma, page 5087, left column, paragraph 6)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Regarding claim 12, the rejection of claim 11 is incorporated. Ma discloses a method, wherein selecting the tunable hyperparameter comprises selecting a default value: “In our case, the data “points” are minibatches of activations from different network layers, and thus have different dimensionality. This suggests that different σ (hyperparameter) values should be used in each kernel. Using the observation that the expected squared distance between random points scales with dimension, these multiple hyperparameters can be approximated by scaling a single σ (default value) with the dimensionality d of points x and y as:
PNG
media_image11.png
63
688
media_image11.png
Greyscale
(used in all our experiments).” (Ma, page 5087, left column, paragraph 6)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Palanisamy to train its network with Ma’s HSIC Bottleneck method, rather than with Palanisamy’s conventional backpropagation method. Ma’s method is able to train networks that cannot be trained with conventional backpropagation, mitigates vanishing and exploding gradient issues found in backpropagation, and potentially allows layers to be trained in parallel, all the while maintaining performance competitive with conventional backpropagation approaches. See Ma, page 5091, left column, paragraphs 3-4.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Palanisamy (SplitEasy: A Practical Approach for Training ML models on Mobile Devices, published 1/29/2021, arXiv:2011.04232v2) in view of Ma (The HSIC Bottleneck: Deep Learning without Back-Propagation, published 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) pp. 5085-5092), and further in view of Iwahashi (METHOD FOR PROCESSING SPEECH SIGNAL USING SUB-CONVERTING FUNCTIONS AND A WEIGHTING FUNCTION TO PRODUCE SYNTHESIZED SPEECH, patented 12/30/1997, Patent No. 5,704,006), and Kageyama (COMPUTER, NEURAL NETWORK CONSTRUCTION METHOD, AND COMPUTER SYSTEM, published 5/21/2020, US 2020/0160118 A1).
Regarding claim 13, the rejection of claim 11 is incorporated. While the aforementioned references fail to disclose the further limitations of the claim, Kageyama discloses a method, wherein selecting the tunable hyperparameter selecting the hyperparameter using trial and error: “During construction of CNNs according to a conventional technique, a learning apparatus sets hyperparameters by trial and error on the basis of the know-how or the like of the designer, or using random numbers, and executes a learning process for attaining a global optimum solution” (Kageyama, [0007])
Kageyama relates to tuning hyperparameters in neural network systems and is analogous to the claimed invention. The existing combination teaches a neural network learning method that relies upon a hyperparameterized kernel. The claimed invention improves upon this method by selecting the kernel hyperparameter using trial and error. Kageyama teaches a method of selecting hyperparameters using trial and error, applicable to the existing combination. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that selecting a hyperparameter using trial and error would lead to the predictable result of evaluating a large set of hyperparameter values, and would improve the known device by determining the best-performing hyperparameter value from among the large set of candidates (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Palanisamy (SplitEasy: A Practical Approach for Training ML models on Mobile Devices, published 1/29/2021, arXiv:2011.04232v2) in view of Ma (The HSIC Bottleneck: Deep Learning without Back-Propagation, published 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) pp. 5085-5092), and further in view of Iwahashi (METHOD FOR PROCESSING SPEECH SIGNAL USING SUB-CONVERTING FUNCTIONS AND A WEIGHTING FUNCTION TO PRODUCE SYNTHESIZED SPEECH, patented 12/30/1997, Patent No. 5,704,006), and Mohanty (AUTOMATED HYPERPARAMETER TUNING IN MACHINE LEARNING ALGORITHMS, filed 6/29/2021, US 2022/0414533 A1).
Regarding claim 14, the rejection of claim 11 is incorporated. While the aforementioned references fail to disclose the further limitations of the claim, Mohanty discloses a method, wherein selecting the parameter comprises selecting the hyperparameter using an automatic tuning procedure: “Illustrative embodiments of the disclosure provide techniques for automated hyperparameter tuning in machine learning algorithms” (Mohanty, [0003]).
Mohanty relates to tuning hyperparameters in neural network systems and is analogous to the claimed invention. The existing combination teaches a neural network learning method that relies upon a hyperparameterized kernel. The claimed invention improves upon this method by automatically selecting the kernel hyperparameter. Mohanty teaches a method of automatically selecting hyperparameters, applicable to the existing combination. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that automatically selecting a hyperparameter would lead to the predictable result of evaluating candidate hyperparameter values without human intervention, and would improve the known device by increasing the speed of hyperparameter selection (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Akhtaruzzaman (HSIC Bottleneck Based Distributed Deep Learning Model for Load Forecasting in Smart Grid With a Comprehensive Survey, published 11/24/2020, IEEE Access Volume 8 pp. 222977-223008) discloses a distributed computing system that uses HSIC bottlenecks for training.
Dean (Large Scale Distributed Deep Networks, published 2012, NIPS 2012) discloses a split learning system where a neural network is partitioned across multiple machines.
Lai (Parallel Calculation Method, Device, Device And Medium Of Natural Language Processing Model, files 1/11/2022, CN 114356578 B) discloses a distributed learning system where different groups of neural network layers are partitioned and distributed across a plurality of computing devices.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron P Gormley whose telephone number is (571)272-1372. The examiner can normally be reached Monday - Friday 12:00 PM - 8:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AG/Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
