DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-2, 7-11, and 16-19 as presented on 11/11/2025 are pending and are examined herein.
Claims 1-2, 7-11, and 16-19 are rejected under 35 USC 103.
Response to Arguments
Applicant’s arguments filed 11/11/2025 regarding the rejection under 35 USC 102 have been fully considered and are persuasive in part. Applicant’s argument that Savic does not teach the video memory is persuasive.
Applicant further argues, see especially page 10, that the transmission is through the network 260 rather than through an electronic device based on a CPU. Examiner respectfully disagrees. [0037] of Savic indicates that the CPU in the host performs the pipeline management, so this is performed through the distributed first trainer. Note that the plain meaning of “through” encompasses “by means of”. Since the CPU based device manages the transmission, the transmission is through or by means of the CPU device. If Applicant were to amend the claim to require transmission of the data first to the CPU based device from the distributed built-in parameter server and then another transmission to the distributed parameter server from the CPU based device, Applicant’s argument might be persuasive, although further consideration of specific claim language would be required to reach a determination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 10-11, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over “Savic” (US 2019/0325302 A1) in view of “Shraga” (US 2021/0342684 A1).
Regarding claim 1, Savic teaches
A method for distributed training a model, comprising: (Savic, Abstract)
acquiring a training sample set from a distributed file system through a data server; (Savic, [0028] describes accessing training data from persistent storage where the persistent storage is distinct both from the host system and from the worker nodes. It is “distributed” at least with respect to the host system and worker nodes.)
acquiring, by a distributed second trainer, each batch of training samples from the data server through a distributed first trainer; (Savic, [0028] indicates that the host system distributes training data from the persistent storage to the worker nodes. The host system corresponds to the claimed distributed first trainer. The worker nodes considered together correspond to the claimed distributed second trainer.)
performing, for each batch of training samples acquired by the distributed first trainer, model training through the distributed second trainer to obtain gradient information, (Savic, [0016, 0029] describes the worker nodes performing training on a minibatch batch to determine gradients of the loss function.)
the distributed first trainer being a first electronic device based on a Central Processing Unit (CPU), the second distributed trainer being a second electronic device based on a graphics processing unit (GPU) such that the distributed first trainer and distributed second trainer are mutually heterogenous; (Savic, [0020] indicates that the host may comprise a CPU and there may be a plurality of hosts. [0004, 0013, 0015] indicates that the worker nodes may comprise GPU devices. The host and workers are heterogeneous at least insofar as the hosts are based on CPUs and the workers are based on GPUs.)
the distributed first trainer comprises a plurality of first trainers, (Savic, [0020] indicates that the host may comprise a plurality of host processors (i.e., first trainers).)
the distributed second trainer comprises a plurality of second trainers, and (Savic, [0005, 0015-0016] indicates that there may be a plurality of worker nodes. The plurality of worker nodes considered together with the NICs, but excluding the “master parameter server” (see [0043-0045]) is being interpreted as the “distributed second trainer” with individual worker nodes corresponding to the “second trainers”.)
the plurality of first trainers and the plurality of second trainers are in communication; (Savic, [0025] indicates that the host processor devices may communicate with the worker nodes (also referred to as accelerator devices).)
updating a target parameter in a distributed built-in parameter server according to the gradient information, wherein the distributed built-in parameter server includes built-in parameter servers, ([0037, 0043] indicates that the worker nodes have Network Interface Cards (NICs) which include a local parameter server which store local model parameters. See also Figure 2. The collection of worker nodes is taken to be the distributed built-in parameter server.)
a ... memory of each second trainer in the distributed second trainer is provided with one built-in parameter server, and ([0028] indicates that the worker nodes may comprise accelerators devices which include a memory for storing parameters.)
...the target parameter is a portion of parameters of an initial model; and ([0037, 0043] indicates that the worker nodes have Network Interface Cards (NICs) which include a local parameter server which store local model parameters. See also Figure 2. [0044] makes it clear that the local parameters are a subset of the total parameters of the model.)
performing, in response to determining that training for a preset number of training samples is completed, a parameter exchange between the distributed built-in parameter server and a distributed parameter server located outside the first and second trainers through the distributed first trainer to perform a parameter update on the initial model until training for the initial model is completed, wherein the parameter exchange comprises: transmitting the updated target parameter in the distributed built-in parameter server to the distributed parameter server through the distributed first trainer, to perform the parameter update on the initial model in the distributed parameter server; and acquiring a new target parameter from the distributed parameter server through the distributed first trainer, and loading the new target parameter to the distributed built-in parameter server. (Savic, [0043-0045] describes performing a global update by aggregating the local parameters from the local parameter servers to a master parameter server and then distributing the generated global parameters (i.e., new target parameter) to the worker nodes. Note interpretation of “second trainers” above as excluding the master parameter server. [0037] indicates that the CPU in the host performs the pipeline management, so this is performed through the distributed first trainer. [0015] indicates that the processes are performed until a convergence criterion is met. [0016] indicates that the training is performed for mini-batches. The size of the mini-batch is being interpreted as the present number of training samples (see also [0028], where the size is set to be M.).)
Savic does not appear to explicitly teach
a video memory
However, Shraga—directed to analogous art--teaches
a video memory ([0068] describes storing parameters of a neural network in VRAM.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Savic by Shraga to store parameters in VRAM because this allows for faster access as described by Shraga at [0068].
Regarding claim 2, the rejection of claim 1 is incorporated herein. Furthermore, Savic teaches
performing a following parameter update operation until the training for the initial model is completed: transmitting, in response to determining that the training for the preset number of training samples is completed, the updated target parameter in the distributed built-in parameter server to the distributed parameter server through the distributed first trainer, to perform the parameter update on the initial model in the distributed parameter server; and (Savic, [0043-0045] describes performing a global update by aggregating the local parameters from the local parameter servers to a master parameter server. [0037] indicates that the CPU in the host performs the pipeline management, so this is performed through the distributed first trainer. [0015] indicates that the processes are performed until a convergence criterion is met. [0016] indicates that the training is performed for mini-batches. The size of the mini-batch is being interpreted as the present number of training samples (see also [0028], where the size is set to be M.).)
acquiring a target parameter for a next parameter update operation in the distributed built-in parameter server from the distributed parameter server through the distributed first trainer. (Savic, [0043-0045] indicates that the master parameter server broadcasts the aggregated parameter set to the other non-master nodes for the next iteration. [0037] indicates that the CPU in the host performs the pipeline management, so this is performed through the distributed first trainer.)
Regarding claim 10, Savic teaches
An electronic device, comprising: at least one processor; and a memory, communicatively connected with the at least one processor, wherein the memory stores an instruction executable by the at least one processor, and the instruction is executed by the at least one processor, to enable the at least one processor to perform operations, comprising: (Savic, Figure 2-3, described at [0039-0050])
The remainder of claim 10 is substantially similar to claim 1. Claim 10 is rejected with the same rationale, mutatis mutandis.
Regarding claim 11, the rejection of claim 10 is incorporated herein. Claim 11 recites substantially similar subject matter to claim 2 and is rejected with the same rationale.
Regarding claim 19, Savic teaches
A non-transitory computer readable storage medium, storing a computer instruction, wherein the computer instruction, when executed by a computer, causes the computer to perform operations, comprising: (Savic, Figure 2-3, described at [0039-0050]. See especially [0049] for the non-transitory aspect.)
The remainder of claim 19 is substantially similar to claim 1. Claim 19 is rejected with the same rationale, mutatis mutandis.
Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over “Savic” (US 2019/0325302 A1) in view of “Shraga” (US 2021/0342684 A1), further in view of “Naumov” (Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems).
Regarding claim 7, the rejection of claim 1 is incorporated herein. Furthermore, Savic teaches
the method further comprises: adjusting a number of machines of a central processing unit in the data server according to a data scale of the training sample set.
Savic does not appear to explicitly teach
the method further comprises: adjusting a number of machines of a central processing unit in the data server according to a data scale of the training sample set.
However, Naumov—directed to analogous art--teaches
the method further comprises: adjusting a number of machines of a central processing unit in the data server according to a data scale of the training sample set. (Naumov, Section V. Discussion on Scale-out Training describes increasing a number of nodes or super-nodes in response to a growth in the amount of input data. As described in section IV. Zion Scale-up Training, the supernodes comprise both CPUs and accelerators, so an increase in the number of supernodes means an increase in the number of associated CPUs as well.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Savic by Naumov because this allows for the system to keep up with the need for increased training throughput as described by Naumov in Section V. Discussion on Scale-out Training.
Regarding claim 16, the rejection of claim 10 is incorporated herein. Claim 16 recites substantially similar subject matter to claim 7 and is rejected with the same rationale.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over “Savic” (US 2019/0325302 A1) in view of “Shraga” (US 2021/0342684 A1), further in view of “Mohan” (US 2022/0156649 A1).
Regarding claim 8, the rejection of claim 1 is incorporated herein. Furthermore, Savic teaches
wherein an information exchange is performed between trainers ([0021] describes performing both inter- and intra-node communication.)
Savic does not appear to explicitly teach
wherein an information exchange is performed between trainers
However, Mohan—directed to analogous art—teaches
through an information queue. (Mohan, Abstract describes performing distributed machine learning training. [0103-0105] describes passing messages between the computing nodes using as message queue.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Savic by Mohan because the distributed training techniques taught by Mohan allow for allow for faster and asynchronous distributed training as described by Mohan at [0076].
Regarding claim 17, the rejection of claim 10 is incorporated herein. Claim 17 recites substantially similar subject matter to claim 8 and is rejected with the same rationale.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over “Savic” (US 2019/0325302 A1) in view of “Shraga” (US 2021/0342684 A1), further in view of “Henry” (US 2021/0286650 A1).
Regarding claim 9, the rejection of claim 1 is incorporated herein. Savic does not appear to explicitly teach
wherein during the model training, computing power between the trainers is adjusted based on a load balancing strategy, to cause the trainers to be matched with each other in computing power.
However, Henry—directed to analogous art--teaches
wherein during the model training, computing power between the trainers is adjusted based on a load balancing strategy, to cause the trainers to be matched with each other in computing power. (Henry, Abstract describes techniques for distributed machine learning. [0033-0035] describes balancing task models (analogous to worker nodes in the combination with Savic) by increasing or decreasing compute allocations.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Savic by Henry because this allows for an improvement in allocation of compute resources as described by Henry at [0004-0005] and [0012].
Regarding claim 18, the rejection of claim 10 is incorporated herein. Claim 18 recites substantially similar subject matter to claim 9 and is rejected with the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Markus A Vasquez whose telephone number is (303)297-4432. The examiner can normally be reached Monday to Friday 10AM to 2PM PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARKUS A. VASQUEZ/Primary Examiner, Art Unit 2121