DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Preliminary Amendment filed on 3/11/2024. Claims 1-15 are pending in the case. Claims 1, 10, and 15 are independent claims.
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 is rejected under 35 U.S.C. § 101 because the claimed invention is directed to non-statutory subject matter. During examination, the claims must be interpreted as broadly as their terms reasonably allow. In re American Academy of Science Tech Center, 367 F.3d 1359, 1369, 70 U.S.P.Q.2d 1827, 1834 (Fed. Cir. 2004). Independent claim 15 recites a “computer program product,” which is not defined by the specification. The broadest reasonable interpretation of a claim drawn to a system covers software per se in view of the ordinary and customary meaning of computer program product, particularly when the specification is silent. Software per se is not a “process,” a “machine,” a “manufacture,” or a “composition of matter” as defined in 35 U.S.C. § 101. Examiner suggests adding a recitation of a “machine readable storage medium” or a “non-transitory machine readable medium.”
Claim Rejections - 35 U.S.C. § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-15 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Chen et al. (“Parallelizing Adam Optimizer with Blockwise Model-Update Filtering,” added to IEEE Xplore 09 April 2020, https://ieeexplore.ieee.org/document/9052983, hereinafter Chen).
As to independent claim 1, Chen discloses a computer-implemented method, comprising:
providing, by a master node (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1), a global model parameter (“Starting from a common initial model parameter
θ
t
-
τ
(
i
n
i
t
)
,” page 3027 section “2.1. BMUF-based framework” lines 4-5) and a global moment parameter (“We can simply set initial Adam buffers for next IBPO as the averaged buffers of local optimizers as follows:
m
t
(
i
n
i
t
)
=
m
-
t
=
1
N
∑
i
=
1
N
m
t
,
i
where
m
t
,
i
and
v
t
,
i
are moment buffers of the
i
-th worker,” page 3028 column right paragraph 1 lines 3-8) to a plurality of worker nodes for a training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1);
receiving, from the plurality of worker nodes, a plurality of local model parameters and a plurality of local moment parameters, the plurality of local model parameters and the plurality of local moment parameters being generated by respective ones of the plurality of worker nodes performing moment-based optimizations in parallel for the training cycle based on the global model parameter and the global moment parameter (“we propose to parallelize Adam with blockwise model-update filtering (BMUF) [6]. BMUF is a general communication efficient distributed optimization framework, where each worker optimized its local model for several steps to get a local model-update first, then local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 6-12);
aggregating the plurality of local model parameters to obtain an aggregated model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and aggregating the plurality of local moment parameters to obtain an aggregated moment parameter (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4);
generating model update information for the training cycle based on the aggregated model parameter and historical model update information for a preceding training cycle (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12);
updating the global model parameter based on the model update information for the training cycle to obtain an updated global model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12);
updating the global moment parameter based on the aggregated moment parameter to obtain an updated global moment parameter compatible with the updated global model parameter (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4); and
providing the updated global model parameter (“Starting from a common initial model parameter
θ
t
-
τ
(
i
n
i
t
)
,” page 3027 section “2.1. BMUF-based framework” lines 4-5) and the updated global moment parameter (“We can simply set initial Adam buffers for next IBPO as the averaged buffers of local optimizers as follows:
m
t
(
i
n
i
t
)
=
m
-
t
=
1
N
∑
i
=
1
N
m
t
,
i
where
m
t
,
i
and
v
t
,
i
are moment buffers of the
i
-th worker,” page 3028 column right paragraph 1 lines 3-8) to the plurality of worker nodes for performing moment-based optimizations in parallel for a succeeding training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1).
As to dependent claim 2, Chen further discloses a method wherein each local model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and each local moment parameter are generated by one of the plurality of worker nodes performing the moment-based optimizations for the training cycle with a predetermined number of mini-batches of training data (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4), and updating the global moment parameter comprises:
determining the number of equivalent mini-batches for the training cycle based on the predetermined number of mini-batches and the number of equivalent mini-batches for the preceding training cycle, the number of equivalent mini-batches for a training cycle representing a converted number of mini-batches used to generate the model update information for the training cycle (“block momentum can compensate per mini-batch’s inadequate contribution to final model-update caused by averaging operation to improve model performance. We define a variable
p
n
to represent the number of equivalent mini-batches required to get
Δ
n
,” page 3028 column left paragraph 1 lines 1-5); and
updating the global moment parameter based on the aggregated moment parameter and the number of equivalent mini-batches for the training cycle (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4).
As to dependent claim 3, Chen further discloses a method wherein updating the global moment parameter comprises:
determining a first weight for the global moment parameter (“
β
1
m
t
-
1
,” page 3028 column right line 2; “
β
2
v
t
-
1
,” page 3028 column right line 3) and a second weight for the aggregated moment parameter (“
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
(
1
-
β
2
)
g
t
,” page 3028 column right line 3) based on an exponential decay rate and the number of equivalent mini-batches for the training cycle (“the assigned weight to
m
t
-
τ
(
i
n
i
t
)
is
β
1
τ
+
η
p
n
. If
η
is set to
1
-
1
N
,
lim
n
→
+
∞
(
τ
+
η
p
n
)
=
N
τ
. So
m
t
-
τ
(
i
n
i
t
)
’s weight decays exponentially as
N
increases, and consequently, its influence on
m
t
(
i
n
i
t
)
will be alleviated,” page 3028 column right line last to page 3029 column left line 3); and
generating a weighted sum of the global moment parameter with the first weight and the aggregated moment parameter with the second weight to obtain the updated global moment parameter (“
m
t
=
β
1
m
t
-
1
+
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
v
t
=
β
2
v
t
-
1
+
(
1
-
β
2
)
g
t
⊙
g
t
,” page 3028 column right line 3).
As to dependent claim 4, Chen further discloses a method wherein updating the global model parameter comprises:
updating the global model parameter based on the model update information for the training cycle to obtain an intermediate updated global model parameter (“
m
-
t
=
β
1
τ
m
t
-
τ
(
i
n
i
t
)
+
∑
i
=
1
τ
β
1
τ
-
i
1
-
β
1
g
-
t
-
τ
+
o
,” page 3028 column right equation 14); and
updating the intermediate updated global model parameter based on the model update information for the training cycle to obtain the updated global model parameter (“
m
t
(
i
n
i
t
)
=
β
1
τ
(
β
1
η
p
n
-
1
)
1
-
β
1
τ
m
t
-
τ
(
i
n
i
t
)
+
1
-
β
1
τ
+
η
p
n
1
-
β
1
τ
m
-
t
,” page 3028 column right equation 18).
As to dependent claim 5, Chen further discloses a method wherein each local model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and each local moment parameter are generated by one of the plurality of worker nodes performing moment-based optimizations for the training cycle with a predetermined number of mini-batches of training data (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4), and updating the global moment parameter comprises:
determining the number of equivalent mini-batches for the training cycle based on the predetermined number of mini-batches and the number of equivalent mini-batches for the preceding training cycle, the number of equivalent mini-batches for a training cycle representing a converted number of mini-batches used to generate the model update information for the training cycle (“block momentum can compensate per mini-batch’s inadequate contribution to final model-update caused by averaging operation to improve model performance. We define a variable
p
n
to represent the number of equivalent mini-batches required to get
Δ
n
,” page 3028 column left paragraph 1 lines 1-5);
determining the number of equivalent mini-batches for the succeeding training cycle based on the predetermined number of mini-batches and the number of equivalent mini-batches for the training cycle (“
p
n
=
η
p
n
-
1
+
τ
,” page 3028 column left equation 4); and
updating the global moment parameter based on the aggregated moment parameter and the number of equivalent mini-batches for the succeeding training cycle (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4).
As to dependent claim 6, Chen further discloses a method wherein updating the global moment parameter comprises:
determining a third weight for the global moment parameter (“
β
1
m
t
-
1
,” page 3028 column right line 2; “
β
2
v
t
-
1
,” page 3028 column right line 3) and a fourth weight for the aggregated moment parameter (“
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
(
1
-
β
2
)
g
t
,” page 3028 column right line 3) based on an exponential decay rate and the number of equivalent mini-batches for the succeeding training cycle (“
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
(
1
-
β
2
)
g
t
,” page 3028 column right line 3); and
generating a weighted sum of the global moment parameter with the third weight and the aggregated moment parameter with the fourth weight to obtain the updated global moment parameter (“
m
t
=
β
1
m
t
-
1
+
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
v
t
=
β
2
v
t
-
1
+
(
1
-
β
2
)
g
t
⊙
g
t
,” page 3028 column right line 3).
As to dependent claim 7, Chen further discloses a method
wherein generating the model update information for the training cycle comprises:
generating first model update information based on the aggregated model parameter and a block learning rate (“
ζ
(
ϴ
-
t
-
ϴ
t
-
τ
(
i
n
i
t
)
)
,” page 3027 column right equation 1);
generating second model update information based on the historical model update information for the preceding training cycle and a block momentum (“
η
Δ
n
-
1
,” page 3027 column right equation 1); and
combining the first model update information and the second model update information to generate the model update information for the training cycle (“
Δ
n
=
η
Δ
n
-
1
+
ζ
(
ϴ
-
t
-
ϴ
t
-
τ
(
i
n
i
t
)
)
,” page 3027 column right equation 1), and
wherein determining the number of equivalent mini-batches for the training cycle comprises:
determining a first number of equivalent mini-batches based on the predetermined number of mini-batches (“
p
1
=
τ
,” page 3028 column left equation 4) and the block learning rate (“We set
ζ
=
1
in this paper, which is a common practice of BMUF,” page 3028 column left lines 3-4);
determining a second number of equivalent mini-batches based on the number of equivalent mini-batches for the preceding training cycle and the block momentum (“
p
n
=
η
p
n
-
1
+
τ
,” page 3028 column left equation 4); and
combining the first number of equivalent mini-batches and the second number of equivalent mini-batches to determine the number of equivalent mini-batches for the training cycle (“
lim
n
→
+
∞
p
n
=
τ
1
-
η
,” page 3028 column left equation 4).
As to dependent claim 8, Chen further discloses a method wherein the block learning rate is set to 1 (“We set ζ = 1 in this paper, which is a common practice of BMUF,” page 3028 column left lines 3-4) and the block momentum is set based on the number of the plurality of worker nodes (“η is set to
1
-
1
N
,” page 3028 column left paragraph 1 line 11).
As to dependent claim 9, Chen further discloses a method wherein the moment-based optimizations comprise Adam optimizations (“we propose to parallelize Adam with blockwise model-update filtering (BMUF) [6]. BMUF is a general communication efficient distributed optimization framework, where each worker optimized its local model for several steps to get a local model-update first, then local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 6-12), the method further comprising:
updating a bias correction term for the Adam optimizations based on the number of equivalent mini-batches for the training cycle (“Since the respective number of mini-batches of
m
t
(
i
n
i
t
)
has been changed, its bias correction term should be changed accordingly,” page 3028 column right paragraph 3 lines 12-14); and
providing the updated bias correction term (“
m
t
(
i
n
i
t
)
=
β
1
τ
(
β
1
η
p
n
-
1
)
1
-
β
1
τ
m
t
-
τ
(
i
n
i
t
)
+
1
-
β
1
τ
+
η
p
n
1
-
β
1
τ
m
-
t
,” page 3028 column right equation 18) to the plurality of worker nodes for performing the Adam optimizations in parallel for a succeeding training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1).
As to independent claim 10, Chen discloses an electronic device, comprising:
a processing unit (“computing node,” page 3029 column right paragraph 2 line 8);
a memory coupled to the processing unit and storing instructions thereon, the instructions, when executed by the processing unit, performing acts (“computing node,” page 3029 column right paragraph 2 line 8) comprising:
providing, by a master node (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1), a global model parameter (“Starting from a common initial model parameter
θ
t
-
τ
(
i
n
i
t
)
,” page 3027 section “2.1. BMUF-based framework” lines 4-5) and a global moment parameter (“We can simply set initial Adam buffers for next IBPO as the averaged buffers of local optimizers as follows:
m
t
(
i
n
i
t
)
=
m
-
t
=
1
N
∑
i
=
1
N
m
t
,
i
where
m
t
,
i
and
v
t
,
i
are moment buffers of the
i
-th worker,” page 3028 column right paragraph 1 lines 3-8) to a plurality of worker nodes for a training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1);
receiving, from the plurality of worker nodes, a plurality of local model parameters and a plurality of local moment parameters, the plurality of local model parameters and the plurality of local moment parameters being generated by respective ones of the plurality of worker nodes performing moment-based optimizations in parallel for the training cycle based on the global model parameter and the global moment parameter (“we propose to parallelize Adam with blockwise model-update filtering (BMUF) [6]. BMUF is a general communication efficient distributed optimization framework, where each worker optimized its local model for several steps to get a local model-update first, then local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 6-12);
aggregating the plurality of local model parameters to obtain an aggregated model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and aggregating the plurality of local moment parameters to obtain an aggregated moment parameter (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4);
generating model update information for the training cycle based on the aggregated model parameter and historical model update information for a preceding training cycle (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12);
updating the global model parameter based on the model update information for the training cycle to obtain an updated global model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12);
updating the global moment parameter based on the aggregated moment parameter to obtain an updated global moment parameter compatible with the updated global model parameter (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4); and
providing the updated global model parameter (“Starting from a common initial model parameter
θ
t
-
τ
(
i
n
i
t
)
,” page 3027 section “2.1. BMUF-based framework” lines 4-5) and the updated global moment parameter (“We can simply set initial Adam buffers for next IBPO as the averaged buffers of local optimizers as follows:
m
t
(
i
n
i
t
)
=
m
-
t
=
1
N
∑
i
=
1
N
m
t
,
i
where
m
t
,
i
and
v
t
,
i
are moment buffers of the
i
-th worker,” page 3028 column right paragraph 1 lines 3-8) to the plurality of worker nodes for performing moment-based optimizations in parallel for a succeeding training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1).
As to dependent claim 11, Chen further discloses a device wherein each local model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and each local moment parameter are generated by one of the plurality of worker nodes performing moment-based optimizations for the training cycle with a predetermined number of mini-batches of training data (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4), and updating the global moment parameter comprises:
determining the number of equivalent mini-batches for the training cycle based on the predetermined number of mini-batches and the number of equivalent mini-batches for the preceding training cycle, the number of equivalent mini-batches for a training cycle representing a converted number of mini-batches used to generate the model update information for the training cycle (“block momentum can compensate per mini-batch’s inadequate contribution to final model-update caused by averaging operation to improve model performance. We define a variable
p
n
to represent the number of equivalent mini-batches required to get
Δ
n
,” page 3028 column left paragraph 1 lines 1-5); and
updating the global moment parameter based on the aggregated moment parameter and the number of equivalent mini-batches for the training cycle (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4).
As to dependent claim 12, Chen further discloses a device wherein updating the global moment parameter comprises:
determining a first weight for the global moment parameter (“
β
1
m
t
-
1
,” page 3028 column right line 2; “
β
2
v
t
-
1
,” page 3028 column right line 3) and a second weight for the aggregated moment parameter (“
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
(
1
-
β
2
)
g
t
,” page 3028 column right line 3) based on an exponential decay rate and the number of equivalent mini-batches for the training cycle (“the assigned weight to
m
t
-
τ
(
i
n
i
t
)
is
β
1
τ
+
η
p
n
. If
η
is set to
1
-
1
N
,
lim
n
→
+
∞
(
τ
+
η
p
n
)
=
N
τ
. So
m
t
-
τ
(
i
n
i
t
)
’s weight decays exponentially as
N
increases, and consequently, its influence on
m
t
(
i
n
i
t
)
will be alleviated,” page 3028 column right line last to page 3029 column left line 3); and
generating a weighted sum of the global moment parameter with the first weight and the aggregated moment parameter with the second weight to obtain the updated global moment parameter (“
m
t
=
β
1
m
t
-
1
+
(
1
-
β
1
)
g
t
,” page 3028 column right line 2; “
v
t
=
β
2
v
t
-
1
+
(
1
-
β
2
)
g
t
⊙
g
t
,” page 3028 column right line 3).
As to dependent claim 13, Chen further discloses a device wherein updating the global model parameter comprises:
updating the global model parameter based on the model update information for the training cycle to obtain an intermediate updated global model parameter (“
m
-
t
=
β
1
τ
m
t
-
τ
(
i
n
i
t
)
+
∑
i
=
1
τ
β
1
τ
-
i
1
-
β
1
g
-
t
-
τ
+
o
,” page 3028 column right equation 14); and
updating the intermediate updated global model parameter based on the model update information for the training cycle to obtain the updated global model parameter (“
m
t
(
i
n
i
t
)
=
β
1
τ
(
β
1
η
p
n
-
1
)
1
-
β
1
τ
m
t
-
τ
(
i
n
i
t
)
+
1
-
β
1
τ
+
η
p
n
1
-
β
1
τ
m
-
t
,” page 3028 column right equation 18).
As to dependent claim 14, Chen further discloses a device wherein each local model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and each local moment parameter are generated by one of the plurality of worker nodes performing moment-based optimizations for the training cycle with a predetermined number of mini-batches of training data (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4), and updating the global moment parameter comprises:
determining the number of equivalent mini-batches for the training cycle based on the predetermined number of mini-batches and the number of equivalent mini-batches for the preceding training cycle, the number of equivalent mini-batches for a training cycle representing a converted number of mini-batches used to generate the model update information for the training cycle (“block momentum can compensate per mini-batch’s inadequate contribution to final model-update caused by averaging operation to improve model performance. We define a variable
p
n
to represent the number of equivalent mini-batches required to get
Δ
n
,” page 3028 column left paragraph 1 lines 1-5);
determining the number of equivalent mini-batches for the succeeding training cycle based on the predetermined number of mini-batches and the number of equivalent mini-batches for the training cycle (“
p
n
=
η
p
n
-
1
+
τ
,” page 3028 column left equation 4); and
updating the global moment parameter based on the aggregated moment parameter and the number of equivalent mini-batches for the succeeding training cycle (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4).
As to independent claim 15, Chen discloses a computer program product comprising executable instructions, the executable instructions, when executed on a device, cause the device to perform acts (“computing node,” page 3029 column right paragraph 2 line 8) comprising:
providing, by a master node (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1), a global model parameter (“Starting from a common initial model parameter
θ
t
-
τ
(
i
n
i
t
)
,” page 3027 section “2.1. BMUF-based framework” lines 4-5) and a global moment parameter (“We can simply set initial Adam buffers for next IBPO as the averaged buffers of local optimizers as follows:
m
t
(
i
n
i
t
)
=
m
-
t
=
1
N
∑
i
=
1
N
m
t
,
i
where
m
t
,
i
and
v
t
,
i
are moment buffers of the
i
-th worker,” page 3028 column right paragraph 1 lines 3-8) to a plurality of worker nodes for a training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1);
receiving, from the plurality of worker nodes, a plurality of local model parameters and a plurality of local moment parameters, the plurality of local model parameters and the plurality of local moment parameters being generated by respective ones of the plurality of worker nodes performing moment-based optimizations in parallel for the training cycle based on the global model parameter and the global moment parameter (“we propose to parallelize Adam with blockwise model-update filtering (BMUF) [6]. BMUF is a general communication efficient distributed optimization framework, where each worker optimized its local model for several steps to get a local model-update first, then local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 6-12);
aggregating the plurality of local model parameters to obtain an aggregated model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12) and aggregating the plurality of local moment parameters to obtain an aggregated moment parameter (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4);
generating model update information for the training cycle based on the aggregated model parameter and historical model update information for a preceding training cycle (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12);
updating the global model parameter based on the model update information for the training cycle to obtain an updated global model parameter (“local model-updates are aggregated and filtered by a historical model-update with a block momentum to update a global model,” page 3027 section “1. INTRODUCTION” paragraph 2 lines 10-12);
updating the global moment parameter based on the aggregated moment parameter to obtain an updated global moment parameter compatible with the updated global model parameter (“From the above analysis,
Δ
n
can be treated as a model-update by processing
p
n
mini-batches serially. So
m
t
(
i
n
i
t
)
and
v
t
(
i
n
i
t
)
compatible with
θ
t
(
i
n
i
t
)
can be treated as updated buffers after processing
η
p
n
mini-batches from
m
-
t
and
v
-
t
,” page 3028 column right paragraph 2 lines 1-4); and
providing the updated global model parameter (“Starting from a common initial model parameter
θ
t
-
τ
(
i
n
i
t
)
,” page 3027 section “2.1. BMUF-based framework” lines 4-5) and the updated global moment parameter (“We can simply set initial Adam buffers for next IBPO as the averaged buffers of local optimizers as follows:
m
t
(
i
n
i
t
)
=
m
-
t
=
1
N
∑
i
=
1
N
m
t
,
i
where
m
t
,
i
and
v
t
,
i
are moment buffers of the
i
-th worker,” page 3028 column right paragraph 1 lines 3-8) to the plurality of worker nodes for performing moment-based optimizations in parallel for a succeeding training cycle (“BMUF-based training framework,” page 3027 section “2.1. BMUF-based framework” line 1).
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure:
US 2023/0281462 A1 disclosing federated learning with momentum
Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
In the interests of compact prosecution, Applicant is invited to contact the examiner via electronic media pursuant to USPTO policy outlined MPEP § 502.03. All electronic communication must be authorized in writing. Applicant may wish to file an Internet Communications Authorization Form PTO/SB/439. Applicant may wish to request an interview using the Interview Practice website: http://www.uspto.gov/patent/laws-and-regulations/interview-practice.
Applicant is reminded Internet e-mail may not be used for communication for matters under 35 U.S.C. § 132 or which otherwise require a signature. A reply to an Office action may NOT be communicated by Applicant to the USPTO via Internet e-mail. If such a reply is submitted by Applicant via Internet e-mail, a paper copy will be placed in the appropriate patent application file with an indication that the reply is NOT ENTERED. See MPEP § 502.03(II).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan Barrett whose telephone number is 571 270 3311. The examiner can normally be reached 9:00am to 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Michelle Bechtold can be reached at 571 431 0762. The fax phone number for the organization where this application or proceeding is assigned is 571 273 8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ryan Barrett/
Primary Examiner, Art Unit 2148
Prosecution Insights