DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Underwood et al. (US Publication Number 2023/0035657, hereinafter “Underwood”) in view Zhang et al. (US Publication Number 2022/0012629, hereinafter “Zhang”) further in view of Jiang et al. (MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUS", April 16, 2024, Proceedings of the 21st UNENIX Symposium on Networked System Design and Implementation, pp 745 – 760, hereinafter “Jiang”).
4. As per claims 1, 8 and 16, Underwood teaches apparatus, medium, and method comprising: a network interface device comprising: a host interface (computer system 600 host interface for coupling the NIC 108/122/124/630, figures 1 and 6); a direct memory access (DMA) circuitry (NIC 630 communicates with 620 moving data into cache 634, where data transmitted from and delivered to user-space buffers without kernel - DMA paragraph 27, and transmit results from buffer instead of pushing the results to host via 112, figure 1, paragraph 31, paragraphs 76 - 79); a network interface to receive, in at least one packet, time data associated with at least one of multiple layers (630, figure 6 coupled to network 608 with communication logic unit 620 for receiving data from nodes in network 608 with data packet handling seen in figure 5b, paragraphs 77 - 79), wherein the multiple layers provide inputs to a collective operation (collective operation figures 3 and 4, paragraphs 37 – 40, node contributes a partial result and the allreduce can combine the results and distribute them to participating processes, 410 – 416).
Underwood does not appear to explicitly disclose a collective operation associated with a large language model (LLM); and circuitry to: based, at least in part, on the time data associated with at least one of multiple layers, identify a first operation of a first layer of the multiple layers as a late completing process (straggler) relative to times to completion of multiple operations of other layers of the multiple layers and based on the first operation being identified as a late completing process, perform a remedial action to adjust at least one configuration of a first device to execute a second operation of the first layer.
However, Zhang discloses circuitry to: based, at least in part, on the time data associated with at least one of multiple layers, identify a first operation of a first layer of the multiple layers as a late completing process (straggler) relative to times to completion of multiple operations of other layers of the multiple layers (figures 1, 3 and 4, learner manager operates by determining training times to perform distributed deep learning to identify late completing process or straggler of the learner based on a table and threshold for the training time, paragraphs 30 – 33) and based on the first operation being identified as a late completing process, perform a remedial action to adjust at least one configuration of a first device to execute a second operation of the first layer (figure 3, modifying a processing aspect of the straggler to reduce future training time of the straggler for performing the distributed deep learning training on a new batch of training data in response to identifying a straggler and the modification being applied to the next unit of work exactly as the claim applies it to a second operation, paragraphs 27 – 33).
Underwood and Zhang are analogous art because they are from the same field of endeavor time sensitive data handling.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Underwood and Zhang before him or her, to modify the packet handling of Underwood to include the structure of Zhang because it would enhance collective operation of the system.
One of ordinary skill would be motivated to make such modification in order to enhance the progressive calculation handling of data (paragraphs 2 – 4). Therefore, it would have been obvious to combine Zhang with Underwood to obtain the invention as specified in the instant claims.
Underwood/Zhang does not appear to explicitly disclose a collective operation associated with a large language model (LLM).
However, Jiang discloses time data associated with at least one of multiple layers (figure 7, page 752, gather latency data to form heat map which allows for characterization of each computational machine) wherein the multiple layers provide inputs to a collective operation associated with a large language model (LLM) (LLM functionality for a performance based metric system, page 745, introduction section to section 3, page 747); and circuitry to: based, at least in part, on the time data associated with at least one of multiple layers, identify a first operation of a first layer of the multiple layers as a late completing process (straggler) relative to times to completion of multiple operations of other layers of the multiple layers (multiple layer seen in table 1 page 753 for the parallel training visualization with relative identification of heat-map showing stragglers and ranked latency therein, figure 7 and table 2, page 754).
Underwood/Zhang and Jiang are analogous art because they are from the same field of endeavor of straggler handling for data processing systems.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Underwood/Zhang and Jiang before him or her, to modify the learning functionality of Underwood/Zhang to include the learning enhancements of Jiang because would allow for deeper recursive handling of data analysis.
One of ordinary skill would be motivated to make such modification in order to enhance efficiency and stability for the data processing tasks (abstract, page 745) Therefore, it would have been obvious to combine Jiang with Underwood/Zhang to obtain the invention as specified in the instant claims.
5. Underwood modified by the teachings of Zhang/Jiang as seen in claim 1 above, as per claim 2, Underwood teaches an apparatus, wherein the first device comprises one or more of: a central processing unit (CPU), graphics processing unit (GPU), general purpose GPU, neural processing unit (NPU), application specific integrated circuit (ASIC), tensor processing unit (TPU), matrix math unit (MMU), memory, cache, or an accelerator (computing system 100, figure 1, with computing node 102-106 has a CPU 100, GPUs 114, multiple DL accelerators, memory devices 112, NIC ALU 116, cache 118, ASIC, paragraph 26).
6. Underwood modified by the teachings of Zhang/Jiang as seen in claim 1 above, as per claim 3, Zhang teaches an apparatus, wherein the at least one configuration of the first device comprises one or more of: memory bandwidth, memory allocation, cache allocation, power allocation to a processor, or processor clock frequency (clock rate and modifying frequency rate of the processors of learners, paragraph 32).
7. Underwood modified by the teachings of Zhang/Jiang as seen in claim 1 above, as per claim 4, Jiang teaches an apparatus, wherein at least one of the time data associated with at least one of multiple layers comprises a time to complete the first operation of the first layer and/or number of floating point operations per second to complete the first operation of the first layer (figure 7, performance analysis tool with execution time to gather latency data of the computation phase across devices and straggler analysis).
8. Underwood modified by the teachings of Zhang/Jiang as seen in claim 1 above, as per claim 5, Jiang teaches an apparatus, wherein the circuitry is to: based on the first operation of the first layer being identified as a late completing process after performance of the remedial action, select a second device and cause a migration of the first operation of the first layer to the second device (section 6.3 with 4.1 the escalation second step taken on hosts that remain slow after measures where hosts took approximately 10% more time to execute for improvement in MFU, section 4.1 where driver submits IP addresses of the nodes to be blocked), wherein the select the second device is based on network bandwidth telemetry provided by at least one other network interface device (section 4,3 replacement host pass through NIC the loopback test measures the bandwidth from RDMA NIC various intra-host endpoints including memory noes and GPU with full-mesh test within the host, section 4.2 RDMA traffic metrics an indicator for network utilization and efficiency).
9. Underwood modified by the teachings of Zhang/Jiang as seen in claim 1 above, as per claim 6, Underwood teaches an apparatus, wherein the host interface is to receive time data associated with a second group of multiple layers, wherein the second group of multiple layers provide inputs to the collective operation (Underwood, figure 6 coupling 630 to 600 by local node with 402…408 arrangement, figure 4) and wherein the circuitry is to: based, at least in part, on the time data associated with at least one of multiple layers and the time data associated with the second group of multiple layers, identify a first operation of a second layer of the multiple layers as a late completing process and based on the first operation of the second layer being identified as a late completing process, perform a remedial action to adjust at least one configuration of a second device to execute a second operation of the second layer (section 5.1, figure 7, analysis is population wide heat map aggregating latency across devices for hosts 0-11 and rank 0-47 simultaneously for distributed data parallelism, tensor parallelism).
10. Underwood modified by the teachings of Zhang/Jiang as seen in claim 1 above, as per claim 7, Underwood teaches an apparatus, wherein the collective operation comprises one or more of: broadcast, AllReduce, reduce, barrier, AllGather, or scatter (broadcast in paragraph 69, AllReduce and AllGather in paragraph 37, barrier in paragraph 60).
11. Underwood modified by the teachings of Zhang/Jiang as seen in claim 8 above, as per claim 9, Underwood teaches a medium, wherein the circuitry comprises one or more of: a central processing unit (CPU), graphics processing unit (GPU), general purpose GPU, neural processing unit (NPU), application specific integrated circuit (ASIC), tensor processing unit (TPU), matrix math unit (MMU), memory, cache, or an accelerator (computing system 100, figure 1, with computing node 102-106 has a CPU 100, GPUs 114, multiple DL accelerators, memory devices 112, NIC ALU 116, cache 118, ASIC, paragraph 26).
12. Underwood modified by the teachings of Zhang/Jiang as seen in claim 8 above, as per claim 10, Underwood teaches a medium, wherein the perform the remedial action to adjust at least one configuration of circuitry comprises increase one or more of: memory bandwidth, memory allocation, cache allocation, power allocation to a processor, processor clock frequency (figure 3 increase applied to straggler frequency rate of stragglers processor while remaining learners directionally decreased), decomposition of an operation to execute on multiple devices, or migration of the operation to a second device (Jiang, sections 4.1 and 6.3 removal of slow host and replacement with healthy one after which training resumes, decomposition is the parallelism where the model is split across GPUs and distributed model layers across multiple devices).
13. Underwood modified by the teachings of Zhang/Jiang as seen in claim 8 above, as per claim 11, Zhang teaches a medium, comprising instructions stored thereon, that if executed by one or more processors, cause the one or more processors to: configure the network interface device to: based on identification of a third operation of the first layer as an early completing operation, perform a second remedial action to adjust at least one configuration of circuitry for performance of the third operation of the first layer, wherein the perform the second remedial action to adjust at least one configuration of circuitry comprises reduce one or more of: memory bandwidth, memory allocation, cache allocation, power allocation to a processor, or processor clock frequency (frequency rate of remaining learners decreased, figure 3, to balance amount of training time by increasing training time for learners not straggling, early finishers throttled down).
14. Underwood modified by the teachings of Zhang/Jiang as seen in claim 8 above, as per claim 12, Jiang teaches a medium, comprising instructions stored thereon, that if executed by one or more processors, cause the one or more processors to: configure the network interface device to: identify the first operation as a late completing operation based on a time to completion of the first operation relative to times to completion of multiple operations of other layers (figure 7, sections 2 pipeline parallelism, section 5.1 analysis tool where heat map reveals minor fraction of machines substantially slower and training efficiency determined by straggler slowest machine ranking).
15. Underwood modified by the teachings of Zhang/Jiang as seen in claim 8 above, as per claim 13, Jiang teaches a medium, wherein the times to completion of multiple operations of other layers comprise an aggregation of the times to completions of the multiple first operations of other layers (section 5.1, latency data of the computation phase across devices and average latency across step visualized as heat map).
16. Underwood modified by the teachings of Zhang/Jiang as seen in claim 8 above, as per claim 12, Underwood teaches a medium, wherein the network interface device is to receive the times to completion of multiple first operations of other layers through a host interface or a network interface (600 with host interface to couple with NIC 630 which coupled to network 608 and communication logic unit 620 receiving the data packet from nodes, figure 6, and both paths seen in figure 1 108 to 110 to 126).
17. Underwood modified by the teachings of Zhang/Jiang as seen in claim 16 above, as per claim 15, Underwood teaches a method, wherein the network interface device comprises one or more of: a network interface controller (NIC), a remote direct memory access (RDMA)-enabled NIC, SmartNIC, router, switch, forwarding element, infrastructure processing unit (IPU), data processing unit (DPU), or edge processing unit (EPU) (NIC, figures 1 and 6, paragraph 26).
18. Underwood modified by the teachings of Zhang/Jiang as seen in claim 16 above, as per claim 17, Underwood teaches a method, wherein the circuitry comprises one or more of: a central processing unit (CPU), graphics processing unit (GPU), general purpose GPU, neural processing unit (NPU), application specific integrated circuit (ASIC), tensor processing unit (TPU), matrix math unit (MMU), memory, cache, or an accelerator (computing system 100, figure 1, with computing node 102-106 has a CPU 100, GPUs 114, multiple DL accelerators, memory devices 112, NIC ALU 116, cache 118, ASIC, paragraph 26).
19. Underwood modified by the teachings of Zhang/Jiang as seen in claim 16 above, as per claim 18, Zhang teaches a method, wherein the at least one configuration of the circuitry comprises one or more of: memory bandwidth, memory allocation, cache allocation, power allocation to a processor, or processor clock frequency (modifying frequency rate of processors of learners, figure 3, Underwood: cache handling in paragraph 30, 118, figure 1).
20. Underwood modified by the teachings of Zhang/Jiang as seen in claim 16 above, as per claim 17, Jiang teaches a method, wherein the times to completion of multiple operations of other layers comprise a number of floating point operations per second to complete the multiple operations of other layers (section 1 and 6.1, table 2 throughput expressed and reported in floating point operation per second MFU ratio observed assuming 100% of peak PFlop).
21. Underwood modified by the teachings of Zhang/Jiang as seen in claim 16 above, as per claim 20, Zhang teaches a method, comprising: identifying a second operation of the first layer as an early completing operation based on a time to completion of the second operation relative to times to completion of multiple second operations of other layers and based on identification of the second operation as an early completing operation, adjusting at least one configuration of circuitry to perform a third operation of the first layer (figures 3 and 4 to determine early completing operation and identify stragglers raising frequency to ensure balancing and increasing training time for early learners, paragraphs 32 – 38).
Conclusion
22. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Hande/Jiang/Srikanthan has teachings of multilayer handling for data packet processing analysis and stragglers therein.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AURANGZEB HASSAN whose telephone number is (571)272-8625. The examiner can normally be reached 7 AM to 3 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henry Tsai can be reached at 571-272-4176. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
AH
/HENRY TSAI/Supervisory Patent Examiner, Art Unit 2184
Prosecution Insights