Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 3, 9, 10, 16 and 17 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 2, 9 and 16 each recite that the parallelizable tasks "are allocated among the plurality of model endpoints according to a target allocation distribution". As amended, independent claims 1, 8 and 15 no longer recite an allocating step. They recite identifying "a target allocation" between "two or more model endpoints" and instructing those endpoints in accordance with it. The dependent claims are unclear in three respects:
Whether "a target allocation distribution" is the same as, a part of, or something different from "the target allocation" of the independent claim.
Which recited step performs the allocating, now that the independent claims recite identifying and instructing.
Whether allocation "among the plurality of model endpoints" requires every endpoint, when the independent claims require only "two or more".
Claims 3, 10 and 17 depend from claims 2, 9 and 16 respectively and are rejected for the same reasons. For purposes of examination, the "target allocation distribution" is interpreted as the "target allocation" identified in the independent claim.
Claim Rejections – Withdrawn
Claim Rejections - 35 U.S.C. 101 (Withdrawn)
The rejection of claims 1-20 under 35 U.S.C. 101 as directed to a judicial exception without significantly more is also withdrawn. The amended independent claims recite instructing the compute resources at two or more model endpoints to perform different subsets of the parallelizable tasks in accordance with the target allocation. That step cannot be performed in the human mind, and it applies the claimed determinations to improve the functioning of the distributed computing system, as described in specification paragraphs [0015] and [0018]-[0019]. The claims therefore integrate the recited exception into a practical application. See MPEP 2106.04(d)(1) and 2106.05(a).
Claim Rejections - 35 U.S.C. 102 (Withdrawn)
The rejection of claims 1-20 under 35 U.S.C. 102(a)(2) as anticipated by Fang (US 2025/0097163 A1) is withdrawn in view of the amendment. Fang allocates request and token quota among client tenancies ([0005], [0098]). It does not teach identifying parallelizable tasks of a workload, identifying a target allocation of those tasks between two or more model endpoints, or instructing those endpoints to perform different subsets of the tasks. New grounds of rejection necessitated by the amendment are set forth below.
Claim Rejections 35 U.S.C 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 5, 7-9, 11, 12, 15, 16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shukla et al. (US 2019/0342379 A1, published Nov. 7, 2019, "Shukla") in view of Fang et al. (US 2025/0097163 A1, "Fang") and White et al. (US 2024/0303127 A1, "White").
Regarding claim 1, Shukla teaches a system for improved utilization of compute hardware distributed among a plurality of servers of a service (workloads are allocated over the servers of a server set, which may be a geographically distributed collection, to reduce overprovisioning and underprovisioning, [0009], [0044], [0058]; the workloads include artificial intelligence applications such as natural language processing, [0002], [0059]), the system comprising:
a processor (processor 304, [0047]);
a consumption determination engine stored in memory and executable to determine a net resource consumption for processing tasks in a workload generated by a client (a workload evaluation 204 determines a service unit cost, a volume of computational resources, for each workload unit of a client's workload, [0011], [0039], [0079] a workload comprises tasks that each consume different resources, [0033], and costs are determined per task and per operation, [0045], [0062]-[0063], [0086], [0090]; the techniques are implemented as instructions stored in memory, [0051], [0054]); and
a load balancer stored in the memory and executable by the processor (the data service provisions and load-balances the server set by method 500, implemented as instructions stored in memory and executed by a processor, [0012], [0051]-[0052]) to:
determine a distribution of available resource capacity in a shared resource pool comprising compute resources at the plurality of servers (a service unit rate capacity 210 and a spare capacity are identified for each server of the shared, multitenant server set, [0010], [0035], [0040]-[0042], FIG. 2);
identify a target allocation of the tasks between two or more servers, the target allocation being based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource pool (a server subset is identified and each server is assigned an allocated service unit rate for a portion of the workload, within that server's capacity, [0015], [0052], FIG. 5; for example, a workload is distributed over two servers that each provide 100 service units per minute, [0012]); and
instruct the compute resources at the two or more servers to perform different subsets of the tasks in accordance with the target allocation (each server receives an allocation request to allocate its computational resources to perform its portion of the workload at the allocated rate, [0013], [0048]-[0050], FIG. 4; each server is a logical endpoint for the portion assigned to it, [0096]).
Shukla does not expressly teach (a) that the servers are model endpoints of a cloud-based service executing instances of a trained machine learning model, with the workload generated by a client application for input to the trained machine learning model, or (b) identifying parallelizable tasks of the workload, such that the target allocation is of the parallelizable tasks and the subsets performed at the endpoints are subsets of the parallelizable tasks. Fang teaches (a). Fang discloses a cloud-based generative AI service in which model servers 132a-m, running on a GPU node pool 130, execute trained machine learning models ([0065], [0077]). A model-launcher in each model server exposes an endpoint for serving inference requests, and incoming requests are load-balanced across multiple model-serving instances ([0083], [0085]). Cloud applications hosted by client tenancies generate the requests that are input to the models ([0039], [0067], [0070]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to apply Shukla's consumption-based and capacity-based allocation to the model endpoints of Fang's cloud-based machine learning service. Shukla states that its techniques apply to artificial intelligence workloads ([0002], [0059]). Fang explains that the GPUs running the models are expensive and costly to leave idle ([0044]). Applying Shukla's allocation to Fang's model servers would predictably reduce both idle and overloaded endpoints (Shukla [0044]).
Shukla as modified in view of White teach (b). Shukla teaches that a workload may be partitioned over the server set ([0004]) and that a task may be partitioned into slices handled by different servers ([0030]). Shukla also distinguishes operations that require tight concurrency from operations with relaxed or no concurrency constraints, which may be completed in no particular sequence ([0063]). White teaches an application job comprising a set of tasks ([0084], [0152]) and determines a resource demand for each task ([0071], [0153], [0157]-[0158]). The tasks of each job are sorted and individually assigned to the first candidate edge system with sufficient assignable capacity, and that system's capacity is updated after each assignment ([0221], [0231]-[0233]). Each task is dispatched to its target edge system ([0154]). Tasks that have no ordering constraint and are individually dispatched to different systems are tasks capable of being performed in parallel. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to identify, among the tasks of the workload, those that can be performed without an ordering dependency, as Shukla recognizes, and to allocate those tasks individually across the endpoints, as White does. Only such tasks can be split across servers without violating sequencing, and White teaches that task-level placement by demand and capacity mitigates overutilization and balances load across systems ([0202]).
Regarding claim 8, the claim recites a method comprising the steps performed by the system of claim 1. Thus, for the same reasons presented with respect to claim 1, claim 8 is unpatentable over Shukla in view of Fang and White.
Regarding claim 15, the claim recites one or more tangible computer-readable storage media encoding processor-executable instructions for executing the computer process of claim 8. Shukla teaches a computer-readable memory device storing instructions that, when executed by a processor, cause a server to perform its techniques ([0053]-[0054], FIG. 6). Thus, for the same reasons presented with respect to claim 1, claim 15 is unpatentable over Shukla in view of Fang and White.
Regarding claim 2, Shukla in view of Fang and White teaches the system of claim 1, wherein the parallelizable tasks of the workload are allocated among the plurality of model endpoints according to a target allocation distribution that is based on a fractional distribution of the available resource capacity across the plurality of model endpoints within the shared resource pool (Shukla: each server's allocated rate for its portion of the workload is set as a share of that server's available service unit rate capacity, so the allocation follows how the available capacity is divided among the servers, [0012], [0042], [0052], [0099]; for example, a workload is divided between two servers according to the 100 service units per minute that each can provide, [0012]).
Regarding claim 4, Shukla in view of Fang and White teaches the system of claim 1, wherein the consumption determination engine is configured to:
determine resource consumption characteristics of the workload (Shukla: the sequence of operations of the workload, the operation types, the volume of data involved and the applicable performance dimensions are examined, [0062]-[0063], [0067]); and
determine the net resource consumption for each of the parallelizable tasks of the workload based on the resource consumption characteristics (Shukla: the per-operation costs are summed to give the service unit cost of each workload unit, and costs are determined for each workload task, [0062], [0086], [0090]; White: a resource demand is determined for each task from data collected about handling the task, including input work statistics, [0155], [0157]-[0158]).
Regarding claim 5, Shukla in view of Fang and White teaches the system of claim 4, wherein the trained machine learning model is a transformer model (Fang: the machine learning model may use a transformer model or a large language model, [0002]) and the net resource consumption for each of the parallelizable tasks is determined based on an identity of the transformer model and a set of inputs to the workload. Fang teaches that the service hosts different models ([0082]). Fang further teaches that the resources a given model needs depend on that model's configuration, its batch size and the number of input tokens it processes ([0099], [0106], [0113]), and that each request comprises one or more tokens ([0041]). Shukla determines consumption from the details of the workload, including the volume of input data ([0063], [0067]).
Because Fang teaches that resource use varies with which model handles a request and with the request's input tokens, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to determine each task's net resource consumption based on the identity of the transformer model and the inputs to the workload. Doing so yields the accurate load estimate that Shukla identifies as necessary to avoid underprovisioning and overprovisioning ([0006]-[0007]).
Regarding claim 7, Shukla in view of Fang and White teaches the system of claim 1, wherein the load balancer determines the distribution of available resource capacity by requesting, from an endpoint discovery mechanism, capacity measurements pertaining to availability of compute resources supporting execution of the model instances at the model endpoints, the endpoint discovery mechanism being configured to retrieve the capacity measurements from the model endpoints. White teaches an orchestrator with a data monitoring and collection module 320 that gathers, directly or through an agent 225 at each edge system, the resources available at each system and its load ([0079]-[0081], [0227]). The scheduler uses the data obtained by that module to assign tasks ([0082]). The module and agents correspond to the claimed endpoint discovery mechanism that retrieves capacity measurements from the endpoints. Fang similarly teaches a metrics service that collects GPU and model-serving metrics for the deployed models ([0084]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, for the load balancer to request the capacity measurements from such a mechanism, so that the distribution of available capacity is determined from current measurements retrieved from the endpoints themselves, which White identifies as informing scheduling decisions ([0201]).
Regarding claim 9, the claim recites substantially the same limitations as claim 2, applied to the method of claim 8. Thus, for the same reasons presented with respect to claim 2, claim 9 is unpatentable over Shukla in view of Fang and White.
Regarding claim 11, the claim recites the limitations of claim 4, applied to the method of claim 8, and further recites requesting capacity measurements pertaining to availability of compute resources supporting execution of the model instances at the model endpoints. The limitations shared with claim 4 are taught for the reasons presented with respect to claim 4. As to requesting capacity measurements, White teaches an orchestrator that obtains available resource and utilization information from each edge system through an agent at that system ([0079], [0081], [0225], [0227]). It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to request such measurements from the endpoints so that the allocation reflects current capacity ([0201]).
Regarding claim 12, the claim recites substantially the same limitations as claim 5, applied to the method of claim 11. Thus, for the same reasons presented with respect to claim 5, claim 12 is unpatentable over Shukla in view of Fang and White.
Regarding claim 16, the claim recites substantially the same limitations as claim 2, applied to the storage media of claim 15. Thus, for the same reasons presented with respect to claim 2, claim 16 is unpatentable over Shukla in view of Fang and White.
Regarding claim 18, the claim recites substantially the same limitations as claim 11, applied to the storage media of claim 15. Thus, for the same reasons presented with respect to claim 11, claim 18 is unpatentable over Shukla in view of Fang and White.
Regarding claim 20, the claim recites substantially the same limitations as claim 5, applied to the storage media of claim 18. Thus, for the same reasons presented with respect to claim 5, claim 20 is unpatentable over Shukla in view of Fang and White.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL NWUHA whose telephone number is (571)272-9367. The examiner can normally be reached Monday-Friday; 7:30 am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kevin Young can be reached at (571) 270-3180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL OBINNA NNAJI NWUHA/Examiner, Art Unit 2194
/KEVIN L YOUNG/Supervisory Patent Examiner, Art Unit 2194
