Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 11 recites the limitation “the distributed training system.” There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 5, 8, 9, 10, 13-16, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Langford (US 2017/0308789) and further in view of Zhang (US 2020/0175370).
Regarding claim 1, Langford teaches: A distributed computing method, comprising:
acquiring a data computing task (¶ 38, “the training data 216 is processed by the algorithm as minibatches 218 of input samples, as discussed below”);
splitting the data computing task to obtain subtasks (¶ 38, “training data 216 is processed by the algorithm as minibatches 218 of input samples, as discussed below, batch selection 224 may include configuring the size of input sample batches or minibatches to balance computation accuracy and execution efficiency according to selected criteria”), deploying the subtasks to computing nodes (¶ 35, “DNN training can be performed by multiple nodes in a parallel manner to reduce the time required for training”), and configuring a parallel mode for each of the computing nodes in a distributed training universal frame (¶ 15, “Neural networks such as DNNs are commonly trained with minibatch-based stochastic gradient descent (SGD). SGD can be parallelized along three dimensions, model parameters, layers, and data (and combinations thereof)”);
optimizing information synchronization efficiency for the computing nodes by using a gradient optimization algorithm or a non-gradient optimization algorithm (¶ 15, “Neural networks such as DNNs are commonly trained with minibatch-based stochastic gradient descent (SGD)”); and
aggregating intermediate results generated by the computing nodes (¶ 126, “aggregating the received partitions at the nodes”), and outputting a final computing result corresponding to the data computing task (¶ 44, “the data analysis engine 206 may use the trained DNN 208 to produce output data 234 from the input data 236”).
Langford does not teach; however, Zhang discloses: configuring a connection manner (¶ 23, “a neighbor list may be constructed for each learner. The neighbor list satisfies connectivity, dissemination, deadlock free properties”) and a communication synchronization manner between the computing nodes (¶ 18, “Asynchronous distributed training of one or more machine learning models may be performed by generating a list of neighbor nodes for each node in a plurality of nodes and creating a first thread for continuous communication”).
It would have been obvious to a person having ordinary skill in the art, at the effective filing date of the invention, to have applied the known technique of configuring a connection manner and a communication synchronization manner between the computing nodes, as taught by Zhang, in the same way to the distributed computing method, as taught by Langford. Both inventions are in the field of distributed computing methods, and combining them would have predictably resulted in “decentralized distributed deep learning in a computing system,” as indicated by Zhang (¶ 1).
Regarding claim 2, Langford teaches: The distributed computing method according to claim 1, wherein the parallel mode comprises a data parallel mode (¶ 15, “SGD can be parallelized along three dimensions, model parameters, layers, and data (and combinations thereof)”), a model parallel mode (¶ 41, “Model striping 228 is also referred to herein as “model parallelism.””), and a hybrid parallel mode (¶ 15, “SGD can be parallelized along three dimensions, model parameters, layers, and data (and combinations thereof)”), wherein the data parallel mode comprises sample-based data parallelism and sample dimension-based data parallelism (¶ 38, “the training data 216 is processed by the algorithm as minibatches 218 of input samples”).
Regarding claim 5, Langford teaches: The distributed computing method according to claim 2, wherein when the parallel mode is the model parallel mode (¶ 41, “Model striping 228 is also referred to herein as “model parallelism.””), the method further comprises: horizontally splitting a distributed computing model or vertically splitting a distributed computing model to adapt to the subtasks (¶ 16, “the output layer of the DNNs or any hidden DNN layer may be processed in parallel across multiple processing units”).
Regarding claim 8, Zhang teaches: The distributed computing method according to claim 6, wherein when the specified connection manner is a decentralized architecture (¶ 68, “A decentralized distributed deep learning service 510 is shown”), constructing the distributed computing system in the specified connection manner comprises: determining workers consisting of the computing nodes (¶ 76, “Each learner (e.g., learner 1 or “node 1” in FIG. 6) has a list of its neighbors”); wherein information exchange between the workers is performed by using a Reduce architecture or a Gossip architecture (¶ 72, “select a neighbor node from the list of neighbor nodes, and/or 4) exchange weights with the local neighbor node and averaging the exchanged weights to generate a weighted vector”), and the distributed computing system is constructed by using the Reduce architecture or the Gossip architecture (¶ 102, “A list of neighbor nodes may be generated for each node in a plurality of nodes, as in block 904. A first thread may be created for continuous communication according to a weight management operation, as in block 906”).
Regarding claim 9, Langford teaches: The distributed computing method according to claim 8, wherein when the distributed computing system adopts the Reduce architecture (¶ 42, “Exchanging 230 can include partitioning the gradient values and performing a distributed all-reduce to provide updates to the gradient values to all the nodes”), each of the workers communicates with all other workers comprised in the workers (¶ 42, “Exchanging 230 is the transmission of gradient values between nodes. This permits model updates to be effectively computed in a data-parallel manner across a large number of nodes”) and transmits local information to all the other workers in a broadcast manner (¶ 74, “at least one of the nodes in the system can transmit the quantized modification values directly to at least one other of the node”).
Regarding claim 10, Zhang teaches: The distributed computing method according to claim 8, wherein when the distributed computing system adopts the Gossip architecture (¶ 18, “generating a list of neighbor nodes for each node in a plurality of nodes and creating a first thread for continuous communication”), each of the workers communicates with its neighboring workers comprised in the workers (¶ 72, “exchange weights with the local neighbor node”).
Regarding claim 13, Zhang teaches: The distributed computing method according to claim 1, wherein aggregating the intermediate results generated by the computing nodes, and outputting the final computing result corresponding to the data computing task comprises: aggregating, by using an ADD-SUM aggregation logic or an integrated aggregation logic, the intermediate results generated by the computing nodes (¶ 126, “aggregating the received partitions at the nodes”), and outputting the final computing result corresponding to the data computing task (¶ 42, “Exchanging 230 can include partitioning the gradient values and performing a distributed all-reduce to provide updates to the gradient values to all the nodes”), wherein the ADD-SUM aggregation comprises a full aggregation logic and a partial aggregation logic (¶ 98, “Node 1 aggregates its own stripe 516 (i.e., stripe 516 of gradients computed by Node 1, indicated by the dotted arrow)”), the full aggregation logic is used for assigning different weights to different computing nodes (¶ 42, “performing a distributed all-reduce to provide updates to the gradient values to all the nodes”), and calculating a weighted sum of the intermediate results generated by all the computing nodes (¶ 98, “Aggregate stripe 534 includes the gradients (or an aggregate, e.g., a sum, thereof) of Subset 1 of the model with respect to training samples 1-900”).
Regarding claim 14, Zhang teaches: The distributed computing method according to claim 1, wherein the data computing task is a data computing task sent by a cloud or another distributed computing device (¶ 54, “cloud computing environment 50 comprises one or more cloud computing nodes 10 with which local computing devices used by cloud consumers”) and received via a network or a data link (¶ 67, “Each of networks 420A, 420B, 420C, and 420D, can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet”).
Regarding claim 15, Langford teaches: The distributed computing method according to claim 5, wherein the distributed computing model comprises a neural network model (¶ 34, “uses algorithms to train deep neural network (DNN) 204 (or a plurality of DNNs, and likewise throughout)”).
Regarding claim 16, Langford teaches: The distributed computing method according to claim 1, wherein the connection manner between the computing nodes comprises a communication topology architecture of the computing nodes in the distributed computing system (¶ 74, “the nodes can communicate in a peer-to-peer topology rather than in a master-slave topology”), and a communication manner between the computing nodes in the communication topology architecture (¶ 97, “Nodes 1-3 exchange data among themselves so that each node, in this example, aggregates the stripes corresponding to a single subset”).
Claims 19 and 20 recite commensurate subject matter as claim 1. Therefore, they are rejected for the same reasons.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Langford and Zhang, as applied above, and further in view of Dirac (US 2015/0379428).
Regarding claim 3, Langford and Zhang do not teach; however, Dirac discloses: deploying each of the subtasks to the computing nodes by means of random sampling with replacement and local shuffling sampling (¶ 209, “training samples 3805A, 3805B and 3805C may be obtained from a larger training set 3802 (e.g., using any of a variety of sampling methodologies such as random sampling with replacement)” and ¶ 520, “shuffling observation records of the first chunk prior to presenting the observation records of the first training set as input to the machine learning model”).
It would have been obvious to a person having ordinary skill in the art, at the effective filing date of the invention, to have applied the known technique of deploying each of the subtasks to the computing nodes by means of random sampling with replacement and local shuffling sampling, as taught by Dirac, in the same way to when the sample-based data parallelism is adopted, deploying the subtasks to the computing nodes, as taught by Langford and Zhang. Both inventions are in the field of distributed machine learning training, and combining them would have predictably resulted in “partitioning for parallelism . . . for performing multiple parallel training operations for a model,” as indicated by Dirac (¶ 146).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Langford and Zhang, as applied above, and further in view of Jiang (US 2021/0150372).
Regarding claim 4, Langford and Zhang do not teach; however, Jiang discloses: when the sample dimension-based data parallel is adopted, and the subtasks comprise one or more dimensions of attributes or features, deploying the subtasks to the computing nodes (¶ 50, “If parallel processing is performed by using N processing subnodes, the feature matrix of the sample is divided into N feature subsets based on columns, where each processing subnode corresponds to one feature subset”) comprises: dividing the subtasks according to the attributes or the features to obtain task samples (¶ 62, “the main processing node divides the feature matrix into N feature subsets based on columns, each feature subset including a plurality of feature columns, all the feature subsets obtained through division forming the feature matrix, and any two feature subsets not intersecting with each other”); and allocating the task samples to the computing nodes corresponding to the task samples (¶ 62, “transmits the N feature subsets to the N processing subnodes, respectively”).
It would have been obvious to a person having ordinary skill in the art, at the effective filing date of the invention, to have applied the known technique of when the sample dimension-based data parallel is adopted, and the subtasks comprise one or more dimensions of attributes or features, deploying the subtasks to the computing nodes comprises: dividing the subtasks according to the attributes or the features to obtain task samples; and allocating the task samples to the computing nodes corresponding to the task samples, as taught by Jiang, in the same way to when the sample-based data parallelism is adopted, deploying the subtasks to the computing nodes, as taught by Langford and Zhang. Both inventions are in the field of distributed machine learning training, and combining them would have predictably resulted in “partitioning for parallelism . . . for performing multiple parallel training operations for a model,” as indicated by Jiang (¶ 146).
Claim(s) 6, 7, 11, 12, 17, and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Langford and Zhang, as applied above, and further in view of Sridharan (US 2019/0205745).
Regarding claim 6, Langford and Zhang do not teach; however, Sridharan discloses: configuring the connection manner and the communication synchronization manner between the computing nodes (¶ 217, “the communication module 2117 includes logic to ensure forward progress of distributed compute operations by enabling asynchronous communication between processing nodes”) comprises: determining whether the data computing task comprises a specified connection manner (¶ 200, “To enable communication, multiple types of low-level communication patterns are used to transfer data between nodes”); constructing a distributed computing system in the specified connection manner when the data computing task comprises the specified connection manner (¶ 226, “transparently adjust communication paths between the worker nodes based on the communication pattern to be performed”), wherein the specified connection manner comprises either a centralized architecture or a decentralized architecture (¶ 190, “update based data parallelism can be performed in a decentralized manner, where the updates are compressed and transferred between nodes”); and parsing the data computing task to obtain the communication synchronization manner (¶ 190, “data parallel training approaches all require a technique of combining results and synchronizing the model parameters between each node”), and configuring the communication synchronization manner between nodes in the distributed computing system according to the communication synchronization manner (¶ 217, “The asynchronous communication enabled by the communication module 2117 allows overlapping compute and communication operations that efficiently interleave to optimize both compute and communication efficiency and throughput”).
It would have been obvious to a person having ordinary skill in the art, at the effective filing date of the invention, to have applied the known technique of configuring the connection manner and the communication synchronization manner between the computing nodes comprises: determining whether the data computing task comprises a specified connection manner; constructing a distributed computing system in the specified connection manner when the data computing task comprises the specified connection manner, wherein the specified connection manner comprises either a centralized architecture or a decentralized architecture; and parsing the data computing task to obtain the communication synchronization manner, and configuring the communication synchronization manner between nodes in the distributed computing system according to the communication synchronization manner, as taught by Sridharan, in the same way to the distributed computing method, as taught by Langford and Zhang. Both inventions are in the field of distributed machine learning training, and combining them would have predictably resulted in “communication optimizations for distributed machine learning,” as indicated by Sridharan (¶ 1).
Regarding claim 7, Sridharan discloses: The distributed computing method according to claim 6, wherein when the specified connection manner is a centralized architecture, constructing the distributed computing system in the specified connection manner comprises: determining workers consisting of the computing nodes and a server consisting of one or a group of server nodes (); wherein, the workers are used for completing a local training task (¶ 381, “a network interface to enable transmission and receipt of configuration data associated with a set of worker nodes, the worker nodes configured to perform distributed training of the neural network”), communicating with the server through a client interface so as to acquire a latest global model parameter (¶ 190, “Parameter averaging trains each node on a subset of the training data and sets the global parameters (e.g., weights, biases)”), and sending local parameters of the workers to the server (¶ 190, “Parameter averaging trains each node on a subset of the training data and sets the global parameters (e.g., weights, biases)”); and the server is used for aggregating the local parameters sent by each of the workers (¶ 190, “Parameter averaging uses a central parameter server that maintains the parameter data”), and updating the global model parameter by using ADD or SUM operations (¶ 190, “sets the global parameters (e.g., weights, biases) to the average of the parameters from each node”).
Regarding claim 11, Sridharan teaches: The distributed computing method according to claim 6, wherein when the communication synchronization manner is synchronous communication, configuring the communication synchronization manner between the nodes in the distributed computing system according to the communication synchronization manner comprises: configuring the communication synchronization manner between the nodes in the distributed computing system according to synchronous communication (¶ 190, “data parallel training approaches all require a technique of combining results and synchronizing the model parameters between each node”), wherein when any computing node in the distributed training system completes the current round of iteration, after waiting for other computing nodes to complete the current round of iteration tasks corresponding to the other computing nodes, all the computing nodes start to process the next round of training iteration tasks (¶ 202, “An allreduce operation 2005 is used to update the weights of each layer for the next forward pass”).
Regarding claim 12, Zhang teaches: The distributed computing method according to claim 6, wherein when the communication synchronization manner is asynchronous communication, configuring the communication synchronization manner between the nodes in the distributed computing system according to the communication synchronization manner comprises: configuring the communication synchronization manner between the nodes in the distributed computing system according to asynchronous communication (claim 1, “performing asynchronous distributed training of one or more machine learning models by generating a list of neighbor nodes for each node in a plurality of nodes”), wherein when any computing node in the distributed training system completes the current round of iteration, the computing node continues to process the next round of training iteration tasks (claim 1, “creating a first thread for continuous communication according to a weight management operation and a second thread for continuous computation of a gradient for each node, wherein one or more variables are shared between the first thread and the second thread”).
Regarding claim 17, Sridharan teaches: The distributed computing method according to claim 6, wherein the method further comprises: when the data computing task does not comprise a specified connection manner (¶ 224, “The process 2300 additionally includes an operation, as shown at block 2303, to determine if the current network topology that interlinks the nodes is known”), the connection manner between the computing nodes is configured in a default connection manner (¶ 224, “If the current network topology is not known, as determined at block 2303, an operation can be performed to initiate a set of sample training iterations, as shown at block 2304. During the sample training iterations, the communication logic can monitor the network latencies between nodes to determine a network topology approximation, as shown at block 2306”).
Claim 21 recites commensurate subject matter as claim 6. Therefore, it is rejected for the same reason.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Matveev (US 2019/0156214) teaches “[e]ach node simulating a set of NN neurons executes a forward-backward pass that calculates or generates new updated weights of links or edges” (¶ 45), which relates to the disclosed exchange of parameters/gradients between nodes in distributed training.
Hamidouche (US 2019/0258924) teaches “[a] complete backward pass generates a batch of updated parameters for all of the layers in the neural network, including updated weight and/or bias values for some or all of the neurons in the neural network” (¶ 15), which relates to the disclosed parameter-server training with multi-node parameter updates.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACOB D DASCOMB whose telephone number is (571)272-9993. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Vital can be reached at (571) 272-4215. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACOB D DASCOMB/ Primary Examiner, Art Unit 2198
Prosecution Insights