DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The preliminary amendment filed on 03/18/2024 has been entered. Claims 1-20 are pending. Claims 1-6, 8-14 have been amended. Claims 15-20 are newly added.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
Step 1
According to the first part of the analysis, in the instant case, claims 1-7 are directed to a method, claims 8-14 are directed to a device, and claims 15-20 are directed to a non-transitory computer readable medium. Each of these claims fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
For claim 1,
Step 2A Prong One
performing fault isolation on a first computing node, wherein the first computing node is a faulty computing node in a plurality of computing nodes, determining a second computing node, other than the plurality of computing nodes in the computing resource pool;
(These steps for performing fault isolation and determining a second computing node are mental processes)
Step 2A Prong Two
A distributed training method for an artificial intelligence (AI) model, applied to an AI platform, comprising: … the AI platform is associated with a computing resource pool, the computing resource pool comprises the plurality of computing nodes for distributed training of an AI model, and each of the plurality of computing nodes performs a training task for the distributed training of the AI model; … and configuring the second computing node that replaces the first computing node to execute a training task.
(These steps for performing distributed training at a high level of generality are considered insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
For claim 2,
Step 2A Prong One
wherein the first computing node is the faulty computing node when
(This step for detecting a faulty computing node is a mental process)
Step 2A Prong Two
the AI platform detects one or more of:
(This step for detecting using an AI platform is considered mere instructions to apply an exception. See MPEP § 2106.05(f))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are mere-instructions to apply an exception recited at a high degree of generality.
For claim 3,
Step 2A Prong One
wherein if the Al platform detects that the hardware fault occurs on the first computing node, and does not detect that the training process corresponding to the training task executed by the first computing node exits, after performing fault isolation on the first computing node, the method further comprises:
(These steps for detecting and fault isolation are mental processes)
Step 2A Prong Two
training process stopping notification indicates the first computing node to stop the training process corresponding to the executed training task.
(This step for stopping training processes based on a notification is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
For claim 4,
Step 2A Prong One
after performing fault isolation on the first computing node, and before the-determining the second computing node, the method further comprises:
(These steps for detecting and fault isolation are mental processes)
Step 2A Prong Two
sending a training process suspension notification to a third computing node that is not faulty in the plurality of computing nodes, and the training process suspension notification indicates the third computing node to suspend a training process corresponding to the training task for the distributed training of the AI model.
(These steps for sending a notification and stopping training processes based on the notification are insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
For claim 5,
Step 2A Prong One
(Claim 5 depends on claim 4, which has been determined to recite abstract ideas including mental processes. Therefore, claim 5 also recites an abstract idea.)
Step 2A Prong Two
wherein the training process suspension notification instructs the third computing node to suspend, after the third computing node completes gradient calculation of the distributed training of the AI model, the training process corresponding to the training task for the distributed training of the AI model.
(These steps for suspending training processes after computing gradients based on a notification is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
For claim 6,
Step 2A Prong One
wherein after the determining the second computing node, the method further comprises:
(This step for determining is a mental process)
Step 2A Prong Two
sending a training continuing notification to the third computing node, wherein the training continuing notification instructs the third computing node to delete the first computing node and add the second computing node in a communication topology in a training framework of the distributed training of the AI model, and to restore the training process corresponding to the training task for the distributed training of the AI model, and the communication topology is used for gradient synchronization of the distributed training of the AI model.
(These steps for sending a notification to prompt restoration of a training process after determining a computing node is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
For claim 7,
Step 2A Prong One
wherein if the second computing node has not been determined, the method further comprises:
(This step for determining is a mental process)
Step 2A Prong Two
sending a training continuing notification to the third computing node, wherein the training continuing notification indicates the third computing node to delete the first computing node in a communication topology in a training framework of the distributed training of the AI model, and to restore the training process corresponding to the training task for the distributed training of the AI model, and the communication topology is used for gradient synchronization of the distributed training of the AI model.
(These steps for sending a notification to prompt restoration of a training process after determining a computing node is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
For claim 8,
Step 2A Prong One
performing fault isolation on a first computing node, wherein the first computing node is a faulty computing node in a plurality of computing nodes, determining a second computing node, other than the plurality of computing nodes in the computing resource pool;
(These steps for performing fault isolation and determining a second computing node are mental processes)
Step 2A Prong Two
A computing device, comprising: a processor: and a memory coupled to the processor to store instructions, which when executed by the processor, cause the computing device to perform operations, the operations comprising: … wherein an artificial intelligence (AI) platform is deployed on the computing device,
(These steps for performing on a generic computer are mere-instruction to apply an exception. See MPEP § 2106.05(f))
the AI platform is associated with a computing resource pool, the computing resource pool comprises the plurality of computing nodes for distributed training of an AI model, and each of the plurality of computing nodes performs a training task for the distributed training of the AI model; … and configuring the second computing node that replaces the first computing node to execute a training task.
(These steps for performing distributed training at a high level of generality are considered insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are mere-instruction to apply an exception and insignificant extra-solution activity recited at a high degree of generality.
For claims 9-14,
Claims 9-14 are device claims that are substantially similar to method claims 2-7, and are rejected using the same reasoning.
For claim 15,
Step 2A Prong One
performing fault isolation on a first computing node, wherein the first computing node is a faulty computing node in a plurality of computing nodes, determining a second computing node, other than the plurality of computing nodes in the computing resource pool;
(These steps for performing fault isolation and determining a second computing node are mental processes)
Step 2A Prong Two
A non-transitory machine-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising: … wherein an artificial intelligence (AI) platform is deployed on a computing device,
(These steps for performing on a generic computer are mere-instruction to apply an exception. See MPEP § 2106.05(f))
the AI platform is associated with a computing resource pool, the computing resource pool comprises the plurality of computing nodes for distributed training of an AI model, and each of the plurality of computing nodes performs a training task for the distributed training of the AI model; … and configuring the second computing node that replaces the first computing node to execute a training task.
(These steps for performing distributed training at a high level of generality are considered insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are mere-instruction to apply an exception and insignificant extra-solution activity recited at a high degree of generality.
For claims 16, 18-20,
Claims 16, 18-20 are non-transitory computer readable medium claims that are substantially similar to method claims 2, 4-6, and are rejected using the same reasoning.
For claim 17,
Step 2A Prong One
wherein if the Al platform detects that the hardware fault occurs on the first computing node, and does not detect that the training process corresponding to the training task executed by the first computing node exits, after performing fault isolation on the first computing node, the method further comprises:
(These steps for detecting and fault isolation are mental processes)
Step 2A Prong Two
sending a training process stopping notification to the first computing node, wherein the training process stopping notification indicates the first computing node to stop the training process corresponding to the executed training task.
(These steps for sending a notification and stopping training processes based on the notification are insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mental processes while the additional elements are insignificant extra-solution activity recited at a high degree of generality.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 8, 15 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Jun-qin Zhang (hereinafter Zhang) (CN 111078480 A, 2020-04-28).
Regarding claim 1, Zhang teaches;
NOTE: Zhang is a Chinese application that has been translated, where the terms ‘executors’ and ‘actuators’ are used interchangeably to indicate distributed training worker nodes.
A distributed training method for an artificial intelligence (AI) model, applied to an AI platform, comprising:
([pg. 8] Fig. 1 is a schematic diagram of the tensiflow distributed training in an embodiment of the present invention.)
performing fault isolation on a first computing node, wherein the first computing node is a faulty computing node in a plurality of computing nodes,
([Abstract] determining whether an abnormal actuator … exists in the plurality of target actuators... stopping the training task of the abnormal actuator)
NOTE: Zhang teaches performing fault isolation on a first computing node (determining that at least a first actuator / node is abnormal / faulty, and isolating the faulty node from the training process), wherein the first computing node is a faulty computing node in a plurality of computing nodes (the faulty node [abnormal actuator] exists in a plurality of nodes [actuators])
the AI platform is associated with a computing resource pool, the computing resource pool comprises the plurality of computing nodes for distributed training of an AI model, and each of the plurality of computing nodes performs a training task for the distributed training of the AI model;
([pg. 8] Fig. 1 is a schematic diagram of the tensiflow distributed training in an embodiment of the present invention. In fig. 1, S1 indicates the first target actuator, S2 indicates the target actuator 1. N target actuators are shared... The first target executor and the N target executors are respectively provided with a plurality of CPUs and a plurality of GPUs… each target executor reads the sample information in the
message queue and executes the training task.)
NOTE: Zhang teaches an AI platform associated with a computing resource pool (a plurality / pool of hardware based actuators used in the distributed training process), the computing resource pool comprises the plurality of computing nodes (actuators can be considered computing nodes, as they perform computations by executing training tasks) for distributed training of an AI model, and each of the plurality of computing nodes performs a training task for the distributed training of the AI model.
determining a second computing node, other than the plurality of computing nodes in the computing resource pool; and configuring the second computing node that replaces the first computing node to execute the training task.
([pg. 9] the training task of the abnormal actuator can be stopped as soon as possible, the alternative actuator is accessed and started, and the alternative actuator seamlessly replaces the abnormal actuator to continue consuming sample data to execute the training task.)
NOTE: Zhang teaches an alternative actuator (second compute node) which is started only when the first compute node (faulty actuator) is removed from the training process. From this, the alternative actuator is not included in the aforementioned plurality of computing nodes because it was not involved in the distributed learning process until this point.
Thus, Zhang teaches determining a second computing node (the alternative actuator), other than the plurality of computing nodes in the computing resource pool (the plurality of actuators currently participating in the training process), and configuring the second computing node that replaces the first computing node to execute the training task (the alternative actuator replaces the abnormal actuator to execute the training task).
Regarding claim 8,
Claim 8 is a device claim that is substantially similar to method claim 1, with two added limitations, which are taught by Zhang;
A computing device, comprising: a processor: and a memory coupled to the processor to store instructions, which when executed by the processor, cause the computing device to perform operations, the operations comprising: … wherein an artificial intelligence (AI) platform is deployed on the computing device,
([pg. 16] the processor 701 is configured to implement the following steps when executing the program stored in the memory 703)
The remaining limitations are taught using the same reasoning from claim 1.
Regarding claim 15,
Claim 15 is a non-transitory computer readable medium claim that is substantially similar to method claim 1, with two added limitations, which are taught by Zhang;
A non-transitory machine-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising: … wherein an artificial intelligence (AI) platform is deployed on a computing device,
([pg. 16] the processor 701 is configured to implement the following steps when executing the program stored in the memory 703 … The Memory may include a Random Access Memory (RAM))
The remaining limitations are taught using the same reasoning from claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 9, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (CN 111078480 A, 2020-04-28) as applied to claims 1, 8, 15 above, and further in view of Suresh K. Thapa et al. (hereinafter Thapa) (US 20220253337 A1, 2022-08-11).
Regarding claim 2, Zhang teaches;
The method according to claim 1, … the first computing node is the faulty computing node when the Al platform detects a fault in the computing node
Using the same reasoning from claim 1
fails to teach but Thapa teaches;
wherein the computing node is the faulty computing node when the platform detects one or more of: a hardware fault occurs on the first computing node, a training process corresponding to the training task executed by the first computing node exits, or the first computing node reports a fault.
([0011] obtain various telemetries associated with the plurality of node processing modules… telemetries can include, … temperatures, voltages, currents of processing resources (e.g., CPUs, GPU, etc.) … compare telemetries … Based on these comparisons, … predict impending faults or failures associated with the plurality of node processing modules… [0016] the plurality of node processing modules 240a-240n can be configured to train a machine learning model)
NOTE: Thapa teaches detecting faulty computing nodes based on associated hardware.
OBVIOUSNESS TO COMBINE THAPA WITH ZHANG:
Thapa is analogous art the present disclosure as it pertains to detecting faulty compute nodes based on associated hardware.
Zhang teaches computing nodes (executors / actuators) implemented in hardware (CPUs, GPUs, as previously taught) and detecting a faulty node (abnormal actuator) and replacing the faulty node so the training task can continue.
Thapa teaches detecting faulty computing nodes based on hardware indicators.
Thapa further states;
([0011] Unlike a software-based scheme, a hardware-based fault detection and analysis scheme can operate at or near real-time, does not take away processor resources, and can be implemented using minimal logical resources (e.g., look up tables) of existing field-programmable gate arrays (FPGAs).)
NOTE: Thapa indicates that hardware-based fault detection saves operation time and computing resources, compared to software-based fault detection.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the hardware-based fault detection scheme taught by Thapa to detect faulty computing nodes in the distributed training platform taught by Zhang to improve operating time and save processor resources.
Regarding claim 9,
Claim 9 is a device claim that is substantially similar to method claim 2, and is rejected using the same reasoning.
Regarding claim 16,
Claim 16 is a non-transitory computer readable medium claim that is substantially similar to method claim 2, and is rejected using the same reasoning.
Claim(s) 3, 10, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (CN 111078480 A, 2020-04-28) in view of Thapa (US 20220253337 A1, 2022-08-11) as applied to claims 2, 9, 16 above, and further in view of Zhicai Zhan et al. (hereinafter Zhan) (CN 110147547 A, 2019-08-20)
Regarding claim 3, Zhang teaches;
wherein if the Al platform detects that the
([pg. 6] whether the abnormal executors exist in the target executors is determined by detecting the reading speed of the sample data by the target executors in the distributed system … the alternative executors are adopted to continue to execute the training task, so that the abnormal condition of the target executors in the online training process can be detected in real time)
NOTE: Zhang teaches the AI platform detecting faults in real time, during the online training process, indicating that the training processes of the faulty node (executor) did not exit.
Thus, Zhang teaches that the Al platform detects that the fault occurs on the first computing node (a detected faulty node), and does not detect that the training process corresponding to the training task executed by the first computing node exits, because the fault detection occurs in real time during training (i.e., before the training task exits).
after performing fault isolation on the first computing node, the method further comprises: … [causing] the first computing node to stop the training process corresponding to the executed training task.
([pg. 6] if an abnormal actuator is detected, ... At the moment, the training task of the abnormal actuator can be stopped as soon as possible)
NOTE: Zhang teaches after performing fault isolation (i.e., determining a faulty compute node / abnormal actuator which is to be excluded from the training process) on the first computing node (the abnormal / faulty actuator), the method further comprises causing the first computing node to stop the training process corresponding to the executed training task (the training task of the abnormal actuator is stopped).
Zhang fails to teach but Thapa teaches;
platform detects that the hardware fault occurs on the first computing node
([0011] obtain various telemetries associated with the plurality of node processing modules… telemetries can include, … temperatures, voltages, currents of processing resources (e.g., CPUs, GPU, etc.) … compare telemetries … Based on these comparisons, … predict impending faults or failures associated with the plurality of node processing modules… [0016] the plurality of node processing modules 240a-240n can be configured to train a machine learning model)
NOTE: Thapa teaches a computing node is the faulty computing node when the platform detects a fault associated with node hardware.
OBVIOUSNESS:
Using the same reasoning from claim 2, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the hardware-based fault detection scheme taught by Thapa to detect faulty computing nodes in the distributed training platform taught by Zhang to improve operating time and save processor resources.
Zhang and Thapa fail to teach but Zhan teaches;
training process stopping notification indicates the first computing node to stop the training process corresponding to the executed training task.
([pg. 8] if so, the management node firstly asynchronously sends interrupt signal to the training node through the distributed training task queue)
NOTE: Zhan teaches sending a training process stopping notification (interrupt signal) which indicates the first computing node to stop the training process corresponding to the executed training task (the training process of the specified training node is interrupted / stopped).
OBVIOUSNESS TO COMBINE ZHAN WITH ZHANG AND THAPA:
Zhan is analogous art to the present disclosure as it pertains to distributed training processes.
Zhang teaches detecting a first faulty node (abnormal executor) during a distributed training process, then stopping the training task of the faulty node. Zhang, however, does not expressly describe the mechanism by which the faulty node is caused to stop its training process.
Additionally, Zhang states;
([pg. 9] if an abnormal actuator … is detected, the abnormal actuator cannot perform normal training basically, which will affect the smooth performance of the whole training task. At the moment, the training task of the abnormal actuator can be stopped as soon as possible)
NOTE: Zhang details that a faulty node (abnormal actuator) in a distributed training platform impacts the performance of the entire training task, so the training task of the faulty node should be stopped quickly, to prevent as much performance degradation as possible.
Zhan teaches a known control mechanism in a distributed training platform that, based on a certain condition, promptly sends a notification (interrupt signal) to a specific node to stop the associated training process.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the control mechanism taught by Zhan to provide a means for promptly stopping the training process associated with a faulty node in the distributed training platform of Zhang in combination with Thapa to preserve the performance of the training task.
Regarding claim 10,
Claim 10 is a device claim that is substantially similar to method claim 3, and is rejected using the same reasoning.
Regarding claim 17,
Claim 17 is a non-transitory computer readable medium claim that is substantially similar to method claim 3, with one added limitation, which is taught by Zhang and Zhan;
Zhang teaches;
[Stopping the training process of] the first computing node
([pg. 6] if an abnormal actuator is detected, ... At the moment, the training task of the abnormal actuator can be stopped as soon as possible)
Zhang fails to teach but Zhan teaches;
sending a training process stopping notification to the first computing node
([pg. 8] if so, the management node firstly asynchronously sends interrupt signal to the training node through the distributed training task queue)
Using the same reasoning from claim 3, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the notification-based control mechanism taught by Zhan to provide a means for promptly stopping the training process associated with a faulty node in the distributed training platform of Zhang in combination with Thapa to preserve the performance of the training task.
The remaining limitations are taught using the same reasoning as claim 3.
Claim(s) 4-5, 11-12, 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (CN 111078480 A, 2020-04-28) as applied to claims 1, 8, 15 above, and further in view of Elastic Horovod (hereinafter Horovod) (‘Elastic Horovod’, 2019).
Regarding claim 4,
Zhang teaches;
after performing fault isolation on the first computing node, and before the-determining the second computing node, the method further comprises:
([pg. 11] if it is determined that ... the first target actuator is abnormal, ..., and at this time, the training tasks of all target actuators in the distributed system need to be stopped... the alternative first target executor is adopted to restart a new training task… [pg. 11] if an abnormal actuator ... is detected, ... At the moment, the training task of the abnormal actuator can be stopped as soon as possible,)
NOTE: Zhang teaches first detecting an abnormal actuator to be excluded from training (fault isolation), then stopping / suspending the training tasks for all other actuators in the system (which includes at least a third, non-faulty / healthy compute node / actuator), and lastly determining the aforementioned second compute node (the aforementioned alternative executor).
Thus, Zhang teaches suspending a training process of a third compute node (suspending training process of a healthy/third actuator) after performing fault isolation on the first computing node (isolating the abnormal actuator), and before the-determining the second computing node (determining the alternative actuator / executor).
a third computing node that is not faulty in the plurality of computing nodes,
([pg. 11] all target actuators in the distributed system)
NOTE: Any of the computing nodes (actuators) participating in the distributed system that are not the faulty computing node can be considered the third computing node in the plurality of computing nodes (plurality of actuators).
([pg. 11] if it is determined that ... the first target actuator is abnormal, ..., at this time, the training tasks of all target actuators in the distributed system need to be stopped)
NOTE: Zhang teaches that the aforementioned third computing node (any non-faulty actuator) suspends a training process corresponding to the training task (all actuators suspend their training process corresponding to the task, including the healthy/third actuators) for the distributed training of the AI model.
Zhang fails to teach but Horovod teaches;
sending a training process suspension notification to a third computing node that is not faulty in the plurality of computing nodes, and the training process suspension notification indicates the third computing node to suspend a training process.
([pg. 2] But if a failure occurs, you may need to redo up to 10 previously processed batches. Elastic Horovod can avoid these rollbacks by performing what we call a graceful removal of a worker. If the driver process discovers that a host has been made available or flagged for removal, it will push a notification to the workers. The next time state.commit() or the more lightweight state.check_host_updates() is called, a HostsUpdatedInterrupt will be raised.)
NOTE: Zhang teaches a sending a notification to all compute nodes / workers currently in the distributed platform (which includes workers that are not faulty) to suspend / interrupt training processes including state.commit() and state.check_host_updates().
OBVIOUSNESS TO COMBINE HOROVOD AND ZHANG:
Horovod is analogous art to the present disclosure because it pertains to distributed training.
Zhang teaches performing fault isolation on nodes (detecting and isolating abnormal actuators), then stopping training processes of computing nodes, and replacing a faulty computing node to continue the training process.
Horovod teaches a known elastic distributed training mechanism which pushes a notification to non-faulty compute nodes (workers) when a compute node is removed or added to suspend / interrupt current training execution while the distributed training membership is updated.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use Horovod notification mechanism in Zhang because doing so would provide a predictable way to coordinate the non-faulty remaining executors during removal of the abnormal executor, thereby avoiding inconsistent training using an outdated distributed training topology.
Regarding claim 5, Zhang teaches;
[instructing] the third computing node to suspend, … the training process corresponding to the training task for the distributed training of the AI model.
([pg. 11] if it is determined that ... the first target actuator is abnormal, ..., at this time, the training tasks of all target actuators in the distributed system need to be stopped)
NOTE: Zhang teaches that the aforementioned third computing node (any non-faulty actuator) suspends a training process corresponding to the training task (all training tasks are stopped) for the distributed training of the AI model.
Zhang fails to teach but Horovod teaches;
wherein the training process suspension notification instructs the third computing node to suspend, … the training process
([pg. 2] If the driver process discovers that a host has been made available or flagged for removal, it will push a notification to the workers. The next time state.commit() or the more lightweight state.check_host_updates() is called, a HostsUpdatedInterrupt will be raised.)
NOTE: Horovod teaches pushing a notification to all computing nodes (workers), which can include a third, non-faulty computing node, to cause training processes (state.commit() and state.check_host_updates()) to be suspended / interrupted.
suspend, after the third computing node completes gradient calculation of the distributed training of the AI model, the training process
[pg. 3]
PNG
media_image1.png
218
613
media_image1.png
Greyscale
PNG
media_image2.png
179
630
media_image2.png
Greyscale
([pg. 2] and call state.check_host_updates() at the end of each batch)
NOTE: In Horovod, the aforementioned training processes (state.commit() and state.check_host_updates()) are suspended after a batch is trained, where training a batch completes gradient calculations of the workers.
Thus, Horovod teaches suspending, after the third computing node completes gradient calculation of the distributed training of the AI model, the training process.
OBVIOUSNESS:
Using the same reasoning from claim 4, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use Khorovod’s notification mechanism in Zhang because doing so would provide a predictable way to coordinate the non-faulty remaining executors during removal of the abnormal executor, thereby avoiding inconsistent training using an outdated distributed training topology.
Regarding claims 11-12,
Claims 11-12 are device claims that is substantially similar to method claims 4-5, respectively, and are rejected using the same reasoning.
Regarding claims 18-19,
Claims 18-19 are non-transitory computer readable medium claims that are substantially similar to method claims 4-5, respectively, and are rejected using the same reasoning.
Claim(s) 6-7, 13-14, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (CN 111078480 A, 2020-04-28) as applied to claims 1, 8, 15 above, and further in view of Xiao-ya Qiao et al. (hereinafter Qiao) (CN 112000473 A, 2020-08-12).
Regarding claim 6, Zhang teaches;
wherein after the determining the second computing node, the method further comprises:
([pg. 9] the training task of the abnormal actuator can be stopped as soon as possible, the alternative actuator is accessed and started, and the alternative actuator seamlessly replaces the abnormal actuator to continue consuming sample data to execute the training task.)
NOTE: Zhang teaches determining a second, alternative computing node (actuator) as a replacement for the faulty computing node (the abnormal actuator).
Zhang fails to teach but Qiao teaches;
sending a training continuing notification to the third computing node, wherein the training continuing notification instructs the third computing node to delete the first computing node and add the second computing node in a communication topology in a training framework of the distributed training of the AI model,
([pg. 9] monitoring the training node corresponding to the training task; determining the fault node, … deleting the fault node … [pg. 11] after the main node receives the registration request of the new training node, the training task management module controls the main node to construct a new communication topology... [pg. 12] when the main node receives the request of reducing the training node, the training task management module controls the main node to construct a new communication topology ...)
NOTE: Qiao teaches a distributed training process that includes deleting a faulty computing node (much like the claimed first compute node), as well as sending requests / notifications to a main node (which can be considered a third, non-faulty node) to delete / reduce training nodes and add new training nodes in a communication topology.
and to restore the training process corresponding to the training task for the distributed training of the AI model,
([pg. 11] notifying all training nodes of the training task according to the new communication topology organization, and executing the training task.)
and the communication topology is used for gradient synchronization of the distributed training of the AI model.
([pg. 11] During the execution of the training task, the primary node deduces the active degree of the course of the training node from the gradient synchronization request after each mini-batch training.)
OBVIOUSNESS TO COMBINE QIAO WITH ZHANG:
Qiao is analogous art to the present disclosure as it pertains to adding and removing computing nodes in a distributed training platform
Zhang teaches replacing faulty training nodes (abnormal actuators) so that distributed training with the remaining non-faulty training nodes can continue, but does not expressly detail how the remaining non-faulty training nodes update their distributed training communication topology after the faulty node is removed and the replacement node is added.
Qiao teaches a known elastic distributed training mechanism in which a new communication topology is constructed when nodes are added or removed (and even teaches removing faulty nodes), the new topology is synchronized, and training task is executed according to the new topology. This allows distributed training to continue when training nodes are removed / added with a consistent communication topology for gradient synchronization.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the topology update and synchronization mechanism of Qiao in the distributed training process of Zhang to allow the remaining non-faulty training nodes to delete the faulty training node, add the replacement, and continue training with a consistent communication topology for gradient synchronization, thereby avoiding gradient synchronization errors caused by an outdated communication topology.
Regarding claim 7, Zhang fails to teach but Qiao teaches;
wherein if the second computing node has not been determined, the method further comprises:
([pg. 9] monitoring the training node corresponding to the training task; determining the fault node, … deleting the fault node …)
NOTE: Qiao teaches a system which deletes faulty nodes instead of adding a replacement node. Thus, a second (replacement) node has not been determined.
sending a training continuing notification to the third computing node, wherein the training continuing notification indicates the third computing node to delete the first computing node in a communication topology in a training framework of the distributed training of the AI model,
([pg. 9] monitoring the training node corresponding to the training task; determining the fault node, … deleting the fault node … [pg. 12] when the main node receives the request of reducing the training node, the training task management module controls the main node to construct a new communication topology ...)
NOTE: Qiao teaches a distributed training process that includes deleting a faulty computing node (much like the claimed first compute node), as well as sending requests / notifications to a main node (which can be considered a third, non-faulty node) to delete / reduce training nodes in a communication topology.
and to restore the training process corresponding to the training task for the distributed training of the AI model,
([pg. 11] notifying all training nodes of the training task according to the new communication topology organization, and executing the training task.)
and the communication topology is used for gradient synchronization of the distributed training of the AI model.
([pg. 11] During the execution of the training task, the primary node deduces the active degree of the course of the training node from the gradient synchronization request after each mini-batch training.)
OBVIOUSNESS:
Using the same reasoning from claim 6, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the topology update and synchronization mechanism of Qiao in the distributed training process of Zhang to allow the remaining non-faulty training nodes to delete the faulty training node and continue training with a consistent communication topology for gradient synchronization, thereby avoiding gradient synchronization errors caused by an outdated communication topology.
Regarding claims 13-14,
Claims 13-14 are device claims that are substantially similar to method claims 6-7, respectively, and are rejected using the same reasoning.
Regarding claim 20,
Claim 20 is a non-transitory computer readable medium claim that is substantially similar to method claim 6, and is rejected using the same reasoning.
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CHAU T NGUYEN/ Primary Examiner, Art Unit 2145
Prosecution Insights