DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on June 30th, 2025 has been entered and Claims 1-20, 24-26 are pending. Applicant’s amendments to the Specification and Claims have overcome every objection and 112(b) rejection previously set forth in the Non-Final Office Action mailed April 4th, 2025. Claims 21-23 have been withdrawn from consideration.
Response to Arguments
Applicant's arguments filed April 4th, 2025 have been fully considered but they are not persuasive.
Applicant’s arguments with respect to claim(s) 1, 10, and 17 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim(s) 1, 8-10, 17, 24-26 is rejected under 35 U.S.C. 103 as being unpatentable over Vivekraja et al US-11941528-B2 in view of Tokui et al. US-20200167657-A1 and Gomez et al, Non-Patent Literature (“The Reversible Residual Network: Backpropagation Without Storing Activations”).
Regarding Claim 1,
An apparatus to be employed in a distributed neural network (NN),
Vivekraja et al. teaches, “the present disclosure relates to neural network processing, and more specifically, to performing a training process of a neural network in a distributed system” (Col 2, lines 8-9). wherein the apparatus comprises: a first worker to: execute a forward training pass of a first node of a distributed NN, further teaches, “A parallel training process (e.g., training processes 330a, 330b, 330p, etc.) can be performed at each worker node. For example, at around the same time, each worker node can perform operation 312 (e.g., 312a, 312b, 312p, etc.), which includes forward propagation operations” (Col. 12, lines 60-65). Vivekraja et al. also discloses exchange of data between nodes. (Col 20, lines 20-32)
However, Vivekraja et al. may not explicitly teach every aspect of
wherein execution of the forward training pass includes generation of a first computational graph (CG) that is based on one or more inputs related to a second node that is processed by a second worker of the distributed NN;
delete, subsequent to the forward training pass of the first node, the CG;
and execute, a backward pass of the first node, wherein execution of the backward pass includes re-generation of at least a portion of the first CG; and
the second worker communicatively coupled with the first worker, wherein the second worker is to provide the one or more input of the inputs related to the second node.
the backward pass to cease gradient flow to prevent a derivative of a loss function from including a dependency identification on the one or more inputs;
Tokui et al. teaches “The graph generator 14 may generate an operation graph (a computational graph) at timing when data is input into the network. The forward propagator 16 may perform an operation on the input data based on a description of definition for forming the network. The description of the definition may be stored in the storage 12. As another example, a network definition descriptor which describes the definition for forming the network may be provided, and the graph generator 14 may generate the graph while executing forward propagation based on the definition of the network described in the network definition descriptor.” (¶0018, lines 1-11, a computational graph (CG) is generated during a forward pass.) Additionally, “The backward propagator 20 may delete the graph for which the backward propagation has been finished. The deletion may be performed by discarding or deleting the graph itself, or may be performed by substantially discarding the graph data by bringing the memory area or the like in which the graph is saved into a state of capable of overwrite save (for example, releasing or deallocating the memory).” (¶0021, lines 8-14, this further maps the limitation where during a backward pass, the CG is deleted and re-generated when needed) and lastly, Tokui teaches, “can transmit and receive the data via a communication line or the like.” (¶0017, lines 11-12, this explicitly teaches that a second worker can be coupled to another worker).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al. and Tokui et al. that an apparatus for training a neural network in a distributed system would include a first worker node to perform a forward pass. With Tokui et al. disclosing the creation and deletion of a computational graph (CG), one of ordinary skill in the art of implementing an apparatus for distributed neural network training would include that during a backward pass, the creation and deletion of computational graphs, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this apparatus for training a neural network in a distributed system.
Gomez et al., teaches the backward pass to cease gradient flow to prevent a derivative of a loss function from including a dependency identification on the one or more inputs; (Section 5.1, paragraph 1, “to construct the backward-pass computation graph without referencing activations computed in the forward pass. While building the backward graph, we reconstruct the input activations (xˆ1, xˆ2) for each block (Equation 8); Second, we apply tf.stop_gradient on the reconstructed inputs to prevent auto-diff from traversing into the re-constructions’ computation graph”).
Gomez et al and Vivekraja et al are both related to the same field of endeavor (i.e., neural network training). In view of the teachings of Gomez it would have been obvious for a person of ordinary skill in the art to apply the teachings of Gomez to Vivekraja et al before the effective filing date of the claimed invention in order achieve memory redundancy and efficiency of training a neural network (Gomez et al., Introduction, paragraph 1-2, “The key architectural innovation behind ResNets was the residual block, which allows information to be passed directly through, making the backpropagated error signals less prone to exploding or vanishing. This made it possible to train networks with hundreds of layers, and this vastly increased depth led to significant performance gains. Nearly all modern neural networks are trained using backpropagation. Since backpropagation requires storing the network’s activations in memory, the memory cost is proportional to the number of units in the network. Unfortunately, this means that as networks grow wider and deeper, storing the activations imposes an increasing memory burden, which has become a bottleneck for many applications”).
Regarding Claim 8, Vivekraja, Tokui, and Gomez teach the apparatus of claim 1.
wherein the apparatus includes a multi-core processor and wherein the first worker is a first core of the multi-core processor and the second worker is a second core of the multi-core processor.
Vivekraja et al. additionally teaches, “In some examples, the host processor 872 can include multiple processing cores. A multi-core processor may include multiple processing units within the same processor.” (Col 26, lines 53-56) and “In some instances, the hardware processor(s) 920 may be a single core processor or a multi-core processor. A multi-core processor may include multiple processing units within the same processor.” (Col 31, lines 4-7, workers can be separate multi-core processors).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al. that an apparatus for training a neural network in a distributed system would include to make each worker a multi-core processor, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this apparatus for training a neural network in a distributed system.
Regarding Claim 9, Vivekraja, Tokui, and Gomez teach the apparatus of claim 1.
wherein the first worker is a first processor of an electronic device, and the second worker is a second processor of the electronic device.
Vivekraja et al. additionally teaches, “The computing resources may include, for example, a neural network hardware accelerator, a general purpose hardware processor, or other suitable computing systems that support the arithmetic operations involved in the training process. Each computing device can communicate, via network 404, with each other device to exchange weight gradients to perform exchange operations 332, and perform update weights operations 316 after exchange operations 332 are completed.” (Col 13, lines 64-67 and Col 14, lines 1-5, workers are separate devices).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al. that an apparatus for training a neural network in a distributed system would include to make each worker an electronic device, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this apparatus for training a neural network in a distributed system.
Regarding Claim 10,
One or more non-transitory computer-readable media comprising instructions that upon execution of the instructions by a first worker of a distributed neural network (NN) training system are to cause the first worker to:
execute a forward training pass of a first node of a distributed NN
Vivekraja et al. teaches, “Storage devices, the DRAM 830, and any other memory component in the host system 800 are examples of computer-readable storage media. Computer-readable storage media are physical mediums that are capable of storing data in a format that can be read by a device such as the host processor 872. Computer-readable storage media can be non-transitory. Non-transitory computer-readable media can retain the data stored thereon when no power is applied to the media. Examples of non-transitory computer-readable media include ROM devices, magnetic disks, magnetic tape, optical disks, flash devices, and solid state drives, among others. As used herein, computer-readable storage media does not include computer-readable communication media. (121) In various examples, the data stored on computer-readable storage media can include program instructions, data structures, program modules, libraries, other software program components, and/or other data that can be transmitted within a data signal, such as a carrier wave or other transmission. The computer-readable storage media can, additionally or alternatively, include documents, images, video, audio, and other data that can be operated on or manipulated through the use of a software program.” (Col 28, lines 51-67 and Col 29, lines 1-5, discloses non-transitory computer-readable media where it can include instructions).
However, Vivekraja et al. may not explicitly teach every aspect of
wherein execution of the forward training pass includes generation of a first computational graph (CG) that is based on inputs related to a second node that is processed by a second worker of the distributed NN,
delete, subsequent to the forward training pass of the first node, the CG and
execute, a backward pass of the first node, wherein execution of the backward pass includes re-generation of at least a portion of the first CG,
the backward pass to cease gradient flow to prevent a derivative of a loss function from including a dependency identification on the one or more inputs;
Tokui et al. teaches “The graph generator 14 may generate an operation graph (a computational graph) at timing when data is input into the network. The forward propagator 16 may perform an operation on the input data based on a description of definition for forming the network. The description of the definition may be stored in the storage 12. As another example, a network definition descriptor which describes the definition for forming the network may be provided, and the graph generator 14 may generate the graph while executing forward propagation based on the definition of the network described in the network definition descriptor.” (¶0018, lines 1-11, a computational graph (CG) is generated during a forward pass.) Additionally, “The backward propagator 20 may delete the graph for which the backward propagation has been finished. The deletion may be performed by discarding or deleting the graph itself, or may be performed by substantially discarding the graph data by bringing the memory area or the like in which the graph is saved into a state of capable of overwrite save (for example, releasing or deallocating the memory).” (¶0021, lines 8-14, this further maps the limitation where during a backward pass, the CG is deleted and re-generated when needed) and lastly, Tokui teaches, “can transmit and receive the data via a communication line or the like.” (¶0017, lines 11-12, this explicitly teaches that a second worker can be coupled to another worker). Vivekraja et al. also discloses exchange of data between nodes. (Col 20, lines 20-32)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al. and Tokui et al. that an apparatus for training a neural network in a distributed system would include instructions stored on non-transitory computer-readable media, where a first worker node to perform a forward pass. With Tokui et al. disclosing the creation and deletion of a computational graph (CG), one of ordinary skill in the art of implementing an apparatus for distributed neural network training would include to execute instructions stored on non-transitory computer-readable media that during a backward pass, have the creation and deletion of computational graphs, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this apparatus for training a neural network in a distributed system.
Gomez et al., teaches the backward pass to cease gradient flow to prevent a derivative of a loss function from including a dependency identification on the one or more inputs; (Section 5.1, paragraph 1, “to construct the backward-pass computation graph without referencing activations computed in the forward pass. While building the backward graph, we reconstruct the input activations (xˆ1, xˆ2) for each block (Equation 8); Second, we apply tf.stop_gradient on the reconstructed inputs to prevent auto-diff from traversing into the re-constructions’ computation graph”).
Gomez et al and Vivekraja et al are both related to the same field of endeavor (i.e., neural network training). In view of the teachings of Gomez it would have been obvious for a person of ordinary skill in the art to apply the teachings of Gomez to Vivekraja et al before the effective filing date of the claimed invention in order achieve memory redundancy and efficiency of training a neural network (Gomez et al., Introduction, paragraph 1-2, “The key architectural innovation behind ResNets was the residual block, which allows information to be passed directly through, making the backpropagated error signals less prone to exploding or vanishing. This made it possible to train networks with hundreds of layers, and this vastly increased depth led to significant performance gains. Nearly all modern neural networks are trained using backpropagation. Since backpropagation requires storing the network’s activations in memory, the memory cost is proportional to the number of units in the network. Unfortunately, this means that as networks grow wider and deeper, storing the activations imposes an increasing memory burden, which has become a bottleneck for many applications”).
Regarding Claim 17,
A method comprising: executing, by a first worker of a distributed neural network (NN), a forward training pass of a first node of the distributed NN,
Vivekraja et al. teaches, “a method of training a neural network in a distributed system, such as distributed system.” (Col 20, lines 1-2) “A parallel training process (e.g., training processes 330a, 330b, 330p, etc.) can be performed at each worker node. For example, at around the same time, each worker node can perform operation 312 (e.g., 312a, 312b, 312p, etc.), which includes forward propagation operations” (Col. 12, lines 60-65). Vivekraja et al. also discloses exchange of data between nodes. (Col 20, lines 20-32)
However, Vivekraja et al. may not explicitly teach every aspect of
wherein execution of the forward training pass includes generation of a
first computational graph (CG) that is based on one or more inputs related to a second node that is processed by a second worker of the distributed NN;
delete, subsequent to the forward training pass of the first node, the CG;
and execute, a backward pass of the first node, wherein execution of the backward pass includes re-generation of at least a portion of the first CG,
the backward pass to cease gradient flow to prevent a derivative of a loss function from including a dependency identification on the one or more inputs
Tokui et al. additionally teaches “The graph generator 14 may generate an operation graph (a computational graph) at timing when data is input into the network. The forward propagator 16 may perform an operation on the input data based on a description of definition for forming the network. The description of the definition may be stored in the storage 12. As another example, a network definition descriptor which describes the definition for forming the network may be provided, and the graph generator 14 may generate the graph while executing forward propagation based on the definition of the network described in the network definition descriptor.” (¶0018, lines 1-11, a computational graph (CG) is generated during a forward pass). Additionally, “The backward propagator 20 may delete the graph for which the backward propagation has been finished. The deletion may be performed by discarding or deleting the graph itself, or may be performed by substantially discarding the graph data by bringing the memory area or the like in which the graph is saved into a state of capable of overwrite save (for example, releasing or deallocating the memory).” (¶0021, lines 8-14, this further maps the limitation where during a backward pass, the CG is deleted and re-generated when needed).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al. and Tokui et al. that a method for training a neural network in a distributed system would include a first worker node to perform a forward pass. With Tokui et al. disclosing the creation and deletion of a computational graph (CG), one of ordinary skill in the art of implementing an apparatus for distributed neural network training would include that during a backward pass, the creation and deletion of computational graphs, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this method for training a neural network in a distributed system.
Gomez et al., teaches the backward pass to cease gradient flow to prevent a derivative of a loss function from including a dependency identification on the one or more inputs; (Section 5.1, paragraph 1, “to construct the backward-pass computation graph without referencing activations computed in the forward pass. While building the backward graph, we reconstruct the input activations (xˆ1, xˆ2) for each block (Equation 8); Second, we apply tf.stop_gradient on the reconstructed inputs to prevent auto-diff from traversing into the re-constructions’ computation graph”).
Gomez et al and Vivekraja et al are both related to the same field of endeavor (i.e., neural network training). In view of the teachings of Gomez it would have been obvious for a person of ordinary skill in the art to apply the teachings of Gomez to Vivekraja et al before the effective filing date of the claimed invention in order achieve memory redundancy and efficiency of training a neural network (Gomez et al., Introduction, paragraph 1-2, “The key architectural innovation behind ResNets was the residual block, which allows information to be passed directly through, making the backpropagated error signals less prone to exploding or vanishing. This made it possible to train networks with hundreds of layers, and this vastly increased depth led to significant performance gains. Nearly all modern neural networks are trained using backpropagation. Since backpropagation requires storing the network’s activations in memory, the memory cost is proportional to the number of units in the network. Unfortunately, this means that as networks grow wider and deeper, storing the activations imposes an increasing memory burden, which has become a bottleneck for many applications”).
Regarding Claim 24, Vivekraja, Tokui, and Gomez teach the apparatus of claim 1. Vivekraja et al. may not explicitly teach every aspect of
wherein the derivative of the loss function corresponds to an error of loss with respect to an output tensor, the error obtained prior to execution of the backward pass
Gomez et al further teaches wherein the derivative of the loss function corresponds to an error of loss with respect to an output tensor, the error obtained prior to execution of the backward pass (Section 3.2, paragraph 1-2, “In the backwards pass, we are given the activations (y1, y2) and their total derivatives (y1, y2) and wish to compute the inputs (x1, x2), their total derivatives (x1, x2), and the total derivatives for any parameters associated with F and G. (See Section 2.1 for our backprop notation.) We do this by combining the reconstruction formulas (Eqn. 8) with the backprop rule (Eqn. 1). The resulting algorithm is given as Algorithm 1.3 By applying Algorithm 1 repeatedly, one can perform backprop on a sequence of reversible blocks if one is given simply the activations and their derivatives for the top layer in the sequence.”)
The motivation for claim 24 is the same motivation for claim 1.
Regarding Claim 25, Vivekraja, Tokui, and Gomez teach the apparatus of claim 1. Vivekraja et al. may not explicitly teach every aspect of
wherein the dependency identification is a dependence of a component of an output on the one or more inputs,
the backward pass to cease gradient flow using a stopgrad operation, the stopgrad operation applied to an output of de-aggregation operation to identify the component of the output.
Gomez et al further teaches wherein the dependency identification is a dependence of a component of an output on the one or more inputs, (Section 2.1, paragraph 2, “Backprop computes the total derivative dC/dvi for each node in the computation graph. This total derivative defines the the effect on C of an infinitesimal change to vi , taking into account the indirect effects through the descendants of vk in the computation graph.”) the backward pass to cease gradient flow using a stopgrad operation, (Section 5.1, paragraph 1, “to construct the backward-pass computation graph without referencing activations computed in the forward pass. While building the backward graph, we reconstruct the input activations (xˆ1, xˆ2) for each block (Equation 8); Second, we apply tf.stop_gradient on the reconstructed inputs to prevent auto-diff from traversing into the re-constructions’ computation graph”) the stopgrad operation applied to an output of de-aggregation operation to identify the component of the output (Section 3.2, “computations in the following way: z1 = x1 + F(x2) z1 = y1 y2 = x2 + G(z1) x2 = y2 − G(z1) (8) y1 = z1 x1 = z1 − F(x2) Even though z1 = y1, the two variables represent distinct nodes of the computation graph, so the total derivatives z1 and y1 are different. In particular, z1 includes the indirect effect through y2, while y1 does not. This splitting lets us implement the forward and backward passes for reversible blocks in a modular fashion. In the backwards pass, we are given the activations (y1, y2) and their total derivatives (y1, y2) and wish to compute the inputs (x1, x2), their total derivatives (x1, x2), and the total derivatives for any parameters associated with F and G. (See Section 2.1 for our backprop notation.) We do this by combining the reconstruction formulas (Eqn. 8) with the backprop rule (Eqn. 1). The resulting algorithm is given as Algorithm 1.”)
The motivation for claim 25 is the same motivation for claim 1.
Regarding Claim 26, Vivekraja, Tokui, and Gomez teach the apparatus of claim 24. Vivekraja et al. may not explicitly teach every aspect of
wherein the at least a portion of the first CG is identified based on an aggregation operation or a de-aggregation operation relating the one or more inputs to the output.
Gomez et al further teaches wherein the at least a portion of the first CG is identified based on an aggregation operation or a de-aggregation operation relating the one or more inputs to the output (Section 3.2, “computations in the following way: z1 = x1 + F(x2) z1 = y1 y2 = x2 + G(z1) x2 = y2 − G(z1) (8) y1 = z1 x1 = z1 − F(x2) Even though z1 = y1, the two variables represent distinct nodes of the computation graph, so the total derivatives z1 and y1 are different. In particular, z1 includes the indirect effect through y2, while y1 does not. This splitting lets us implement the forward and backward passes for reversible blocks in a modular fashion. In the backwards pass, we are given the activations (y1, y2) and their total derivatives (y1, y2) and wish to compute the inputs (x1, x2), their total derivatives (x1, x2), and the total derivatives for any parameters associated with F and G. (See Section 2.1 for our backprop notation.) We do this by combining the reconstruction formulas (Eqn. 8) with the backprop rule (Eqn. 1). The resulting algorithm is given as Algorithm 1.”)
The motivation for claim 26 is the same motivation for claim 1.
Claim(s) 2-7, 11-16, 18-20 is rejected under 35 U.S.C. 103 as being unpatentable over Vivekraja et al in view of Tokui et al and Gomez et al and further in Chen et al. US-20210042620-A1
Regarding Claim 2, Vivekraja, Tokui, and Gomez teach the apparatus of claim 1.
Vivekraja et al. additionally teaches, “Method 600 starts in step 602, in which the neural network processor of a first worker node of distributed system 400 performs backward propagation computations for the second neural network layer to generate second layer data gradients and second layer weight gradients. The backward propagation computations can be of a first batch. Step 602 can be performed after the first batch forward propagation operations of both the first neural network layer and the second neural network layer have been completed, and input data gradients (din of FIG. 3A) has been generated. The first batch backward propagation computations can be performed based on either the input data gradients, or data gradients output by a higher neural network layer.” (Col 20, lines 20-32, explicitly teaches that a first worker processes inputs based on a backward pass of a second worker).
However, Vivekraja et al. may not explicitly teach every aspect of
wherein the first worker is further to:
identify, based on the backward pass, an alteration to an input of the inputs related to the second node of the distributed NN;
Chen et al. teaches, “The system 100 can insert communication primitives, e.g., instructions, at each composite layer 103 that when executed by a computing device 106, can cause the device 106 to exchange data, e.g., an output activation or an output gradient, to another computing device assigned to a neighboring composite layer.” Continued at, ”the computing device 106a can communicate with the second computing device 106b through instructions 201 that cause a first data exchange 201a to be performed between devices in which the first computing device 106a sends the output activation of the last layer of boundary layers of the first composite layer 103a to the first layer of boundary layers of the second composite layer 103b as an input, and a second data exchange 201b to be performed between the same two devices in a manner that the second computing device 106b sends the output gradient of the first layer of boundary layers of the second composite layer 103b to the last layer of boundary layers of the first composite layer 103a as an input. Similarly, as shown in FIG. 2, the system can insert instructions 203 for communications between the second computing device 106b and the third computing device 106c to allow a third data exchange 203a of an output activation from the composite layer 103b to the composite layer 103c, and a fourth data exchange 203b of an output gradient from the composite layer 103c to the composite 103b.” (¶0070, lines 1-6 and ¶0072, workers have a set of instructions (identify) when data is exchanged to update input values, indicating facilitate the alteration).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al., Tokui, and Chen et al. that an apparatus for training a neural network in a distributed system would include a first worker node to perform a forward pass and process inputs based on a backward pass of a second worker. Vivekraja et al. teaches that the data is related to a second node. With Chen et al. disclosing workers having a set of instructions when data is exchanged, to update input values, one of ordinary skill in the art of implementing an apparatus for distributed neural network training would include a first worker to identify, based on a backward pass of a second worker, where the workers have a set of instructions to update or alter the input values, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this apparatus for training a neural network in a distributed system.
Regarding Claim 3, Vivekraja, Tokui, and Gomez teach the apparatus of claim 1.
Vivekraja et al. additionally teaches, “Following the generation of input data gradients din by loss gradient operation 304, a backward propagation operation 306 can be performed for each neural network layer. For example, a backward propagation operation 306n can be performed at highest layer n, a backward propagation operation 306b can be performed at layer 2, a backward propagation operation 306a can be performed at layer 1. A backward propagation operation at a neural network layer can be based on the weights of that neural network layer, the data gradient input to that neural network layer, as well as the input to the forward propagation operation of that layer. For example, for layer n, backward propagation operation 306n can receive, as inputs, weights wn, input data outn−1 (from forward propagation operation at neural network layer n−1), and input data gradient din.” (Col 11, lines 5-19, input and weight values are from a second node during a backward pass).
However, Vivekraja, Tokui, and Gomez may not explicitly teach every aspect of
wherein the inputs related to the second node include an input value and a weight value of the second node.
Chen et al. teaches, “The system 100 can insert communication primitives, e.g., instructions, at each composite layer 103 that when executed by a computing device 106, can cause the device 106 to exchange data, e.g., an output activation or an output gradient, to another computing device assigned to a neighboring composite layer.” Continued at, ”the computing device 106a can communicate with the second computing device 106b through instructions 201 that cause a first data exchange 201a to be performed between devices in which the first computing device 106a sends the output activation of the last layer of boundary layers of the first composite layer 103a to the first layer of boundary layers of the second composite layer 103b as an input, and a second data exchange 201b to be performed between the same two devices in a manner that the second computing device 106b sends the output gradient of the first layer of boundary layers of the second composite layer 103b to the last layer of boundary layers of the first composite layer 103a as an input. Similarly, as shown in FIG. 2, the system can insert instructions 203 for communications between the second computing device 106b and the third computing device 106c to allow a third data exchange 203a of an output activation from the composite layer 103b to the composite layer 103c, and a fourth data exchange 203b of an output gradient from the composite layer 103c to the composite 103b.” (¶0070, lines 1-6 and ¶0072, workers have a set of instructions (identify) when data is exchanged to update input values, indicating facilitate the alteration). It is obvious that during the facilitate the alteration process, the data exchanged are input and weight values.
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention given the teachings of Vivekraja et al., Tokui, and Chen et al. that an apparatus for training a neural network in a distributed system would include a first worker node to perform a forward pass and process inputs and weights based on a backward pass of a second worker. With Chen et al. disclosing workers having a set of instructions when data is exchanged, to update input and weight values, one of ordinary skill in the art of implementing an apparatus for distributed neural network training would include a first worker to identify, based on a backward pass of a second worker, where the workers have a set of instructions to update or alter the input and weight values, in order to achieve memory redundancy and efficiency. One would therefore be motivated to combine these teachings as in doing so would create this apparatus for training a neural network in a distributed system.
Regarding Claim 4, Vivekraja, Tokui, and Gomez with Chen teach the apparatus of claim 3.
wherein the input value and the weight value are provided by the second worker to the first worker.
Vivekraja et al. additionally teaches, “Following the generation of input data gradients din by loss gradient operation 304, a backward propagation operation 306 can be performed for each neural network layer. For example, a backward propagation operation 306n can be performed at highest layer n, a backward propagation operation 306b can be performed at layer 2, a backward propagation operation 306a can be performed at layer 1. A backward propagation operation at a neural network layer can be based on the weights of that neural network layer, the data gradient input to that neural network layer, as well as the input to the forward propagation operation of that layer. For example, for layer n, backward propagation operation 306n can receive, as inputs, weights wn, input data outn−1 (from forward propagation operation at neural network layer n−1), and input data gradient din.” (Col 11, lines 5-19, input and weight values are from a second node during a backward pass).
Further in view of Chen et al. teaches, ”the computing device 106a can communicate with the second computing device 106b through instructions 201 that cause a first data exchange 201a to be performed between devices in which the first computing device 106a sends the output activation of the last layer of boundary layers of the first composite layer 103a to the first layer of boundary layers of the second composite layer 103b as an input, and a second data exchange 201b to be performed between the same two devices in a manner that the second computing device 106b sends the output gradient of the first layer of boundary layers of the second composite layer 103b to the last layer of boundary layers of the first composite layer 103a as an input. Similarly, as shown in FIG. 2, the system can insert instructions 203 for communications between the second computing device 106b and the third computing device 106c to allow a third data exchange 203a of an output activation from the composite layer 103b to the composite layer 103c, and a fourth data exchange 203b of an output gradient from the composite layer 103c to the composite 103b.” (¶0072, the exchange is from a second worker to a first worker).
Regarding Claim 5, Vivekraja, Tokui, and Gomez with Chen teach the apparatus of claim 3.
wherein the input value is provided by the second worker to the first worker, and the weight value is retrieved from a memory communicatively coupled with the first worker.
Vivekraja et al. teaches, “Following the generation of input data gradients din by loss gradient operation 304, a backward propagation operation 306 can be performed for each neural network layer. For example, a backward propagation operation 306n can be performed at highest layer n, a backward propagation operation 306b can be performed at layer 2, a backward propagation operation 306a can be performed at layer 1. A backward propagation operation at a neural network layer can be based on the weights of that neural network layer, the data gradient input to that neural network layer, as well as the input to the forward propagation operation of that layer. For example, for layer n, backward propagation operation 306n can receive, as inputs, weights wn, input data outn−1 (from forward propagation operation at neural network layer n−1), and input data gradient din.” (Col 11, lines 5-19, input and weight values are from a second node during a backward pass). Further in view of Chen et al. additionally teaches, “Each computing device 106 can have access to a common memory 120, or have its own memory independent to each other.” (¶0067, line 1-3, first and second workers can retrieve data from common memory).
Regarding Claim 6, Vivekraja, Tokui, and Gomez with Chen teach the apparatus of claim 3.
wherein the inputs related to the second node are altered by providing, by the first worker, an indication that the second worker is to alter the input value.
Vivekraja et al. teaches, “Method 600 starts in step 602, in which the neural network processor of a first worker node of distributed system 400 performs backward propagation computations for the second neural network layer to generate second layer data gradients and second layer weight gradients. The backward propagation computations can be of a first batch. Step 602 can be performed after the first batch forward propagation operations of both the first neural network layer and the second neural network layer have been completed, and input data gradients (din of FIG. 3A) has been generated. The first batch backward propagation computations can be performed based on either the input data gradients, or data gradients output by a higher neural network layer.” (Col 20, lines 20-32, explicitly teaches that a first worker processes inputs based on a backward pass of a second worker). Further in view Chen et al. teaches, “The system 100 can insert communication primitives, e.g., instructions, at each composite layer 103 that when executed by a computing device 106, can cause the device 106 to exchange data, e.g., an output activation or an output gradient, to another computing device assigned to a neighboring composite layer.” Continued at, ”the computing device 106a can communicate with the second computing device 106b through instructions 201 that cause a first data exchange 201a to be performed between devices in which the first computing device 106a sends the output activation of the last layer of boundary layers of the first composite layer 103a to the first layer of boundary layers of the second composite layer 103b as an input, and a second data exchange 201b to be performed between the same two devices in a manner that the second computing device 106b sends the output gradient of the first layer of boundary layers of the second composite layer 103b to the last layer of boundary layers of the first composite layer 103a as an input. Similarly, as shown in FIG. 2, the system can insert instructions 203 for communications between the second computing device 106b and the third computing device 106c to allow a third data exchange 203a of an output activation from the composite layer 103b to the composite layer 103c, and a fourth data exchange 203b of an output gradient from the composite layer 103c to the composite 103b.” (¶0070, lines 1-6 and ¶0072, workers have a set of instructions (identify) when data is exchanged to update input values, indicating facilitate the alteration). First and second workers may interchangeably provide indications regarding when to update the values.
Regarding Claim 7, Vivekraja, Tokui, and Gomez with Chen teach the apparatus of claim 3.
wherein the data related to the second node is altered by providing, by the first worker, an indication that the second worker is to alter the weight value.
For the same reasons of claim 6, Vivekraja et al. teaches, “Method 600 starts in step 602, in which the neural network processor of a first worker node of distributed system 400 performs backward propagation computations for the second neural network layer to generate second layer data gradients and second layer weight gradients. The backward propagation computations can be of a first batch. Step 602 can be performed after the first batch forward propagation operations of both the first neural network layer and the second neural network layer have been completed, and input data gradients (din of FIG. 3A) has been generated. The first batch backward propagation computations can be performed based on either the input data gradients, or data gradients output by a higher neural network layer.” (Col 20, lines 20-32, explicitly teaches that a first worker processes inputs based on a backward pass of a second worker).
Further in view Chen et al. teaches, “The system 100 can insert communication primitives, e.g., instructions, at each composite layer 103 that when executed by a computing device 106, can cause the device 106 to exchange data, e.g., an output activation or an output gradient, to another computing device assigned to a neighboring composite layer.” Continued at, ”the computing device 106a can communicate with the second computing device 106b through instructions 201 that cause a first data exchange 201a to be performed between devices in which the first computing device 106a sends the output activation of the last layer of boundary layers of the first composite layer 103a to the first layer of boundary layers of the second composite layer 103b as an input, and a second data exchange 201b to be performed between the same two devices in a manner that the second computing device 106b sends the output gradient of the first layer of boundary layers of the second composite layer 103b to the last layer of boundary layers of the first composite layer 103a as an input. Similarly, as shown in FIG. 2, the system can insert instructions 203 for communications between the second computing device 106b and the third computing device 106c to allow a third data exchange 203a of an output activation from the composite layer 103b to the composite layer 103c, and a fourth data exchange 203b of an output gradient from the composite layer 103c to the composite 103b.” (¶0070, lines 1-6 and ¶0072, workers have a set of instructions (identify) when data is exchanged to update input values, indicating facilitate the alteration). First and second workers may interchangeably provide indications regarding when to update the values.
Regarding Claim 11, Vivekraja, Tokui, and Gomez teach the apparatus of claim 10.
The one or more non-transitory computer-readable media of claim 10, wherein the instructions are further to:
Vivekraja et al. teaches, “Storage devices, the DRAM 830, and any other memory component in the host system 800 are examples of computer-readable storage media. Computer-readable storage media are physical mediums that are capable of storing data in a format that can be read by a device such as the host processor 872. Computer-readable storage media can be non-transitory. Non-transitory computer-readable media can retain the data stored thereon when no power is applied to the media. Examples of non-transitory computer-readable media include ROM devices, magnetic disks, magnetic tape, optical disks, flash devices, and solid state drives, among others. As used herein, computer-readable storage media does not include computer-readable communication media. (121) In various examples, the data stored on computer-readable storage media can include program instructions, data structures, program modules, libraries, other software program components, and/or other data that can be transmitted within a data signal, such as a carrier wave or other transmission. The computer-readable storage media can, additionally or alternatively, include documents, images, video, audio, and other data that can be operated on or manipulated through the use of a software program.” (Col 28, lines 51-67 and Col 29, lines 1-5, discloses non-transitory computer-readable media where it can include instructions). identify, by the first worker, based on the backward pass, an alteration to an input of the inputs related to the second node of the distributed NN; Further teaches, “Method 600 starts in step 602, in which the neural network processor of a first worker node of distributed system 400 performs backward propagation computations for the second neural network layer to generate second layer data gradients and second layer weight gradients. The backward propagation computations can be of a first batch. Step 602 can be performed after the first batch forward propagation operations of both the first neural network layer and the second neural network layer have been completed, and input data gradients (din of FIG. 3A) has been generated. The first batch backward propagation computations can be performed based on either the input data gradients, or data gradients output by a higher neural network layer.” (Col 20, lines 20-32, explicitly teaches that a first worker processes inputs based on a backward pass of a second worker).
However, Vivekraja et al. may not explicitly teach every aspect of ‘facilitate the alteration’ or to identify that an update of inputs and weights are needed.
and
facilitate, by the first worker, the alteration.
Further in view Chen et al. teaches, “The system 100 can insert communication primitives, e.g., instructions, at each composite layer 103 that when executed by a computing device 106, can cause the device 106 to exchange data, e.g., an output activation or an output gradient, to another computing device assigned to a neighboring composite layer.” Continued at, ”the computing device 106a can communicate with the second computing device 106b through instructions 201 that cause a first data exchange 201a to be performed between devices in which the first computing device 106a sends the output activation of the last layer of boundary layers of the first composite layer 103a to the first layer of boundary layers of the second composite layer 103b as an input, and a second data exchange 201b to be performed between the same two devices in a manner that the second computing device 106b sends the output gradient of the first layer of boundary layers of the second composite layer 103b to the last layer of boundary layers of the first composite layer 103a as an input. Similarly, as shown in FIG. 2, the system can insert instructions 203 for communications between the second computing device 106b and the third computing device 106c to allow a third data exchange 203a of an output activation from the composite layer 103b to the composite layer 103c, and a fourth data exchange 203b of an output gradient from the composite layer 103c to the composite 103b.” (¶0070, lines 1-6 and ¶0072, workers have a set of instructions (identify) when data is exchanged to update input values, indicating facilitate the alteration). First and second workers may interchangeably provide indications regarding when to update the values.
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention give