DETAILED ACTION
This action is responsive to the application filed on 08/05/2026. Claims 1-20 are pending and have been examined. This action is Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Argument 1: The applicant argues that the amended claims recite specific machine-learning training, latency prediction, and inference-serving operations requiring measured machine-execution data, rather than merely mathematical concepts or evaluations practically performable mentally. Alternatively, any exception is integrated into a practical application because the specified training data, model inputs, predictions, and configuration-selection operations address costly benchmarking and batch-size variation, with claim 8 additionally requiring execution at the selected configuration. Relying on Enfish, McRO, Desjardins, and USPTO guidance, The applicant requests evaluation of each claim as a whole without oversimplifying its technical limitations and further argues that the ordered combination provides significantly more under Step 2B because the rejection does not establish its conventionality or otherwise satisfy the required evidentiary standard.
Response to Argument 1: The examiner has considered the argument set forth above. In light of the amendments and considering the remarks, the rejection for claims 1-20 under 35 U.S.C 101 are now withdrawn.
Argument 2: The applicant argues that the cited combinations fail to teach the amended training and prediction requirements of independent claims 1, 8, and 16, particularly the use of latency data across multiple ML models and computing configurations together with model parameters and FLOP counts. Romero registers and profiles already-trained model variants rather than training the claimed predictor; Ali estimates service times across memory/batch-size configurations through regression and separately employs an analytical optimizer; and Kotini provides load testing and recommendation metrics. Further, the applicant contends that combining these disclosures still leaves the claimed training relationships missing, identifies specification paragraphs [0060]-[0061] as amendment support, and asserts that the dependent claims are allowable through their dependencies, with some additionally containing independently patentable subject matter.
Response to Argument 2: The examiner has considered the arguments set forth above, however it is not persuasive. The applicant’s arguments regarding the amended training limitations of independent claims 1, 8, and 16 have been fully considered but are moot in part, because the new grounds of rejection, necessitated by amendment, rely on Justus and Li, which were not applied in the prior action, for the training subject matter specifically challenged. The applicant also contends that Romero does not train a predictor, that Ali’s regression covers only memory and batch-size configurations, and that Kotini merely load-tests endpoints; however, these arguments attack the references individually, whereas the rejection rests on what the combined teachings would have suggested to one of ordinary skill in the art (see In re Keller, 642 F.2d 413, 426 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 1097 (Fed. Cir. 1986)). In the combination, Romero supplies the inference-serving framework that registers models, characterizes their floating point workload, and selects among hardware platforms and batch sizes; Ali supplies a regression model, trained on measured batch service times, that predicts latency for configurations that were never profiled and passes those predictions to an optimizer that chooses the resource configuration and batch size; Justus supplies the amended training data, namely a single learned predictor trained on execution times recorded from many randomly generated networks run on several GPU types, using layer hyperparameters, batch size, and hardware features as inputs, and shown to predict execution times even for a GPU type not seen during training; and Li supplies FLOPs as a predictor input, using latency measurements from 1000 neural architectures across six platforms. Kotini, relied on for claims 1-7 and 10, shows per-instance cost and latency results displayed in a user interface under user-set cost and latency importance levels. The combination is reasonable because every reference addresses the same problem as the instant application, which is estimating how long a model will take to run on candidate hardware so that a cost-effective configuration can be chosen without benchmarking every option, and each secondary reference expressly states the benefit of its technique: Ali reduces profiling time by more than 97%, Justus generalizes to previously unseen models and hardware, and Li achieves high accuracy in end-to-end latency prediction. Incorporating the cross-model, cross-hardware predictor of Justus, with the FLOPs features of Li, into the Romero and Ali serving pipeline is therefore the use of known techniques to improve a similar system in the same way, yielding predictable results (see KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 417 (2007)). Finally, Applicant’s general statement that some dependent claims independently recite patentable subject matter does not point out how the language of those claims distinguishes the references (see 37 CFR 1.111(b)), and the dependent claims fall with the independent claims for the reasons above. Thus, claims 1-20 remain rejected under 35 U.S.C. 103 as set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over the NPL reference “INFaaS: Automated model-less inference serving.” by Romero et al. (referred herein as Romero) in view of the NPL reference “Batch: Machine learning inference serving on serverless platforms with adaptive batching.” by Ali et al. (referred herein as Ali) in view of the NPL reference “Predicting the Computational Cost of Deep Learning Models” by Justus et al. (referred herein as Justus) in view of the NPL reference “Predicting Inference Latency of Neural Architectures on Mobile Devices” by Li et al. (referred herein as Li) further in view of the NPL reference “Improved ML model deployment using Amazon SageMaker Inference Recommender” by Kotini et al. (referred herein as Kotini).
Regarding claim 1, Romero teaches:
A method comprising: obtaining an input to a prediction machine learning model, the input indicating a set of model parameters associated with a machine learning model and a number of floating point operations (FLOPS) associated with performing inferencing operations using the machine learning model, ([Romero, page 400, sec 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID).” AND [Romero, page 397, sec 1] “and the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets the model (modelBinary) in serialized format to be the same as the set of model parameters associated with a machine learning model because they are both directed to data defining a trained machine learning model, and that the computational requirements of 0.48 to 24 GFLOPS to be the same as the number of floating point operations (FLOPS) associated with performing inferencing operations because they are both directed to the floating point workload of executing a model for inference.)
determining a set of computing instances capable of executing the machine learning model and a set of batch sizes associated with inferencing requests; ([Romero, page 398, sec 1] “These systems require developers to select model-variants, batch sizes, instance/hardware types, and statically-defined autoscaling configurations, for meeting application requirements.” AND [Romero, page 398, sec 2.1] “A model-variant is a version of a model defined by the following aspects: (a) model architecture (e.g., ResNet50, VGG16), (b) programming framework, (e.g., TensorFlow, PyTorch, Caffe2, MXNet), (c) model graph optimizers (e.g., TensorRT, Neuron, TVM, XLA [72]), (d) hyperparameters (e.g., optimizing for batch size of 1, 4, 8, or 16), and (e) hardware platforms (e.g., Haswell or Skylake CPUs, V100 or T4 GPUs, FPGA, and accelerators, such as Inferentia [18], TPU [49], Catapult [32], NPU [7]).” AND [Romero, page 401, sec 3.2] “From the registered models, the Variant-Generator generates model-variants optimized for different batch sizes, hardware, and hardware-specific parameters (e.g., number of cores on Inferentia)”, wherein the examiner interprets the hardware platforms for which model-variants are generated to be the same as the set of computing instances capable of executing the machine learning model because they are both directed to computing resources on which the model is run, and that the batch sizes of 1, 4, 8, or 16 to be the same as the set of batch sizes associated with inferencing requests because they are both directed to the number of queries processed together during inference.)
determining a set of weighted sum values associated with the set of computing instances based on the latency information; and ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j , this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant” AND [Romero, page 403, sec 4.2.1] “Large values of λ place more weight on minimizing loading latency to meet SLOs when the query load is unpredictable or spiky. Small values of λ place more weight on minimizing the hardware cost when the query load is more stable.”, wherein the examiner interprets the cost for an action, computed as the sum of the hardware cost and the λ-weighted loading latency of each variant, to be the same as the weighted sum values associated with the set of computing instances because they are both directed to per-resource values formed by weighting and adding a latency term and a cost term.)
Romero does not teach causing the prediction machine learning model to output latency information for the set of computing instances and the set of batch sizes, the latency information indicating an interval of time to process an inferencing request of a batch size of the set of batch sizes by a computing instance of the set of computing instances;…wherein the prediction machine learning model has been trained based on a training dataset including latency information obtained from a plurality of computing instances executing a plurality of machine learning models, model parameters associated with the plurality of machine learning models, and numbers of FLOPs associated with the plurality of machine learning models;…causing an indication of the set of weighted sum values associated with the set of computing instances to be displayed in a user interface.
Ali teaches causing the prediction machine learning model to output latency information for the set of computing instances and the set of batch sizes, the latency information indicating an interval of time to process an inferencing request of a batch size of the set of batch sizes by a computing instance of the set of computing instances; ([Ali, page 5, sec III-A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory. BATCH derives the batch service time model using multivariable polynomial regression [44]” AND [Ali, page 7, sec V] “To reduce the profiling time, the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression.” AND [Ali, page 8, sec VI-B] “To determine the mean of the service time distribution with as few experiments as possible for the various memory/batch size configurations, BATCH uses a multivariable regression model that is trained with a few configurations”, wherein the examiner interprets the multivariable regression model that is trained with a few configurations to be the same as the prediction machine learning model because they are both directed to a model trained on measured data to estimate unmeasured performance values, and that the amounts of allocated memory of the memory/batch size configurations to be the same as the set of computing instances because they are both directed to compute resource configurations that execute the inference. The examiner further interprets the estimated batch service time to be the same as the latency information indicating an interval of time to process an inferencing request of a batch size because they are both directed to the time needed to process a batch of inference requests of a given size.)
Romero and Ali do not teach wherein the prediction machine learning model has been trained based on a training dataset including latency information obtained from a plurality of computing instances executing a plurality of machine learning models, model parameters associated with the plurality of machine learning models,…and numbers of FLOPs associated with the plurality of machine learning models;…causing an indication of the set of weighted sum values associated with the set of computing instances to be displayed in a user interface.
Justus teaches wherein the prediction machine learning model has been trained based on a training dataset including latency information obtained from a plurality of computing instances executing a plurality of machine learning models, model parameters associated with the plurality of machine learning models, ([Justus, page 4, sec V-A] “We construct a random selection of these atomic operations embedded within the simplest network we can produce. Each of these atomic operations is then executed either using forward or forwards and backwards passes and the execution times are recorded from multiple executions. The feature set for the atomic operation along with the execution timings are then used to train a fully connected feed forward network.” AND [Justus, page 3, sec IV-A] “These relate to those features of a particular layer within the neural network and in particular to the hyperparameters related to that layer.” AND [Justus, page 4, sec IV-C] “Hardware features are those describing both the GPU card(s) used and the system these cards are housed within” AND [Justus, page 5, Table I] “SPECIFICATION OF HARDWARE TESTED” AND [Justus, page 6, sec VII] “We then test our deep learning predictor when it has been trained on data from multiple GPU types. In this second case we use the trained model to predict execution times for a GPU type not seen as part of the training process.”, wherein the examiner interprets the fully connected feed forward network trained on the feature sets and execution timings to be the same as the prediction machine learning model trained based on a training dataset because they are both directed to a learned model fit to measured execution-time data. The examiner further interprets the execution timings recorded for the randomly selected networks on the multiple GPU types to be the same as latency information obtained from a plurality of computing instances executing a plurality of machine learning models because they are both directed to measured execution times of multiple neural networks on multiple hardware resources, and that the layer hyperparameter features to be the same as model parameters associated with the plurality of machine learning models because they are both directed to values defining the architecture of each model.)
Romero, Ali, and Justus do not teach and numbers of FLOPs associated with the plurality of machine learning models;…causing an indication of the set of weighted sum values associated with the set of computing instances to be displayed in a user interface.
Li teaches and numbers of FLOPs associated with the plurality of machine learning models; ([Li, page 106, sec 4.3.2] “We built a synthetic dataset including 1000 neural architectures sampled from this NAS space. For each neural architecture, we collected training measurements on 6 mobile platforms (Table 1), for a total of 90 scenarios” AND [Li, page 106, Table 3] “Input height (width), input channel, output height (width), stride, kernel height (width), filters, input size, output size, kernel size, FLOPs”, wherein the examiner interprets the FLOPs feature computed for the operations of each of the 1000 neural architectures to be the same as the numbers of FLOPs associated with the plurality of machine learning models because they are both directed to floating point operation counts of multiple models used as inputs for training a latency predictor.)
Romero, Ali, Justus, and Li do not teach causing an indication of the set of weighted sum values associated with the set of computing instances to be displayed in a user interface.
Kotini teaches causing an indication of the set of weighted sum values associated with the set of computing instances to be displayed in a user interface. ([Kotini, page 11] [Results image on page 11] AND [Kotini, page 8] “You can visualize Inference Recommender job results in Amazon SageMaker Studio by choosing Inference Recommender under Deployments in the navigation pane.”, wherein the examiner interprets the per-instance recommendation results produced under the user-adjusted cost, latency, and throughput importance levels to be the same as the indication of the weighted sum values associated with the set of computing instances because they are both directed to instance-level results reflecting a user-weighted combination of cost and latency, and the visualizing the job results in Amazon SageMaker Studio to be the same as displaying in a user interface because they are both directed to presenting results on a graphical interface.)
Romero, Ali, Justus, Li, Kotini, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference based on latency and cost.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model-variant selection and autoscaling technique disclosed by Romero to include the “multivariable regression model that is trained with a few configurations” disclosed by Ali. One would be motivated to do so to efficiently obtain batch service times for the various configurations without exhaustively profiling every configuration, as suggested by Ali ([Ali, page 8, sec VI-B] “Results show that BATCH can reduce the profiling time by more than 97% compared to exhaustively profiling all system configurations.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the execution timings and features used to “train a fully connected feed forward network” disclosed by Justus. One would be motivated to do so to effectively predict execution times for previously unseen models and hardware, as suggested by Justus ([Justus, page 3, sec III] “Our work aims to fill this gap by providing a prediction framework that is capable of dealing with these non-linearities, providing accurate results while also generalising to previously unseen neural network models or hardware.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the “FLOPs” features disclosed by Li. One would be motivated to do so to accurately predict end-to-end inference latency, as suggested by Li ([Li, Abstract] “achieves high accuracy in end-to-end latency predictions”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the ability to “visualize Inference Recommender job results in Amazon SageMaker Studio” disclosed by Kotini. One would be motivated to do so to efficiently enable a user to compare candidate configurations and choose one that satisfies latency and cost requirements, as suggested by Kotini ([Kotini, page 8] “You can use these metrics to analyze and find the best configurations that satisfy latency, concurrency, and cost requirements of your ML application.”).
Regarding claim 2, Romero, Ali, Justus, Li, and Kotini teach The method of claim 1 (see rejection of claim 1).
Romero further teaches:
wherein the method further comprises selecting a first computing instance of the set of computing instances to execute the machine learning model, where based at least in part on a first weighted sum value of the set of weighted sum values. ([Romero, page 402, sec 4.1] “When a query arrives, INFaaS’ Dispatcher invokes the getVariant method of model-variant selection policy to choose a variant (Algorithm 1). For this case, the input to getVariant is the query’s requirements, and the output is the variant and worker to serve the query.” AND [Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j, this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant: Cost(δi j) = Ci j(δi j +λT load i j max(δi j,0))”, wherein the examiner interprets “choose a variant” and “the output is the variant and worker to serve the query” to be the same as selecting a first computing instance of the set of computing instances to execute the machine learning model because they are both directed to choosing a computing resource, including a worker/hardware-associated variant, to serve an inference query using a machine learning model. The examiner further interprets “the total cost of all the chosen scaling actions” and “the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant” to be the same as a first weighted sum value of the set of weighted sum values because they are both directed to a computed sum-based value used to evaluate and choose among resource-selection options based at least in part on latency and cost information.)
Regarding claim 3, Romero, Ali, Justus, Li, and Kotini teach The method of claim 2 (see rejection of claim 2).
Romero further teaches:
wherein the first weighted sum value is greater than at least one other weighted sum value of the set of weighted sum values. ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions.” AND [Romero, page 403, sec 4.2.1] “Thus, our objective function representing the total cost for all the variants is: ∑i, j Cost(δi j).” AND [Romero, page 399, sec 2.2] “In the first case (QPS = 10 and SLO = 300ms), though all variants can meet the QPS and SLO, using two instances of Variant A is the cheapest choice (8× cheaper than using Variant C). In the second case, the load remains unchanged, but due to the stricter SLO, Variant B becomes the cheapest choice (5.3× cheaper than using Variant C).” AND [Romero, page 399, Table 3] “Cheapest configuration (in #instances) of variants from Table 2 to meet the QPS and SLO; last column shows total cost.” AND [Romero, page 406, sec 6.1] “INFaaS achieved higher performance (1.3× higher throughput) and resource utilization (5.6× higher GPU utilization), and lower SLO violations (2× lower) and cost (1.23× lower) compared to the baselines.”, wherein the examiner interprets the total cost computed for each candidate variant configuration by the objective function to be the same as the set of weighted sum values because they are both directed to per-configuration values that combine cost and latency terms and are compared to choose a configuration. The examiner further interprets the selection, under the stricter 50 ms SLO, of the Variant B configuration with a total cost of 3 over the Variant A configuration with a total cost of 2, which can no longer meet the SLO, to be the same as the first weighted sum value being greater than at least one other weighted sum value of the set of weighted sum values because they are both directed to the selected configuration having a computed value that exceeds the value of at least one other candidate configuration, which is set aside because it does not satisfy the latency requirement. The examiner also notes that INFaaS achieving lower SLO violations and lower cost compared to the baselines confirms that these computed values are used to choose among candidate configurations.)
Regarding claim 4, Romero, Ali, Justus, Li, and Kotini teach The method of claim 2 (see rejection of claim 2).
Ali further teaches wherein the method further comprises training the prediction machine learning model using a training dataset including a set of metrics obtained by at least causing the set of computing instances to execute a set of machine learning models performing inferencing operations. ([Ali, page 7, sec VI] “Here, we evaluate the accuracy of the multivariable polynomial model for the estimation of the batch service time (see Section III-A). BATCH is validated experimentally on AWS Lambda” AND [Ali, page 7, sec VI-A] “We evaluate BATCH with two ML Serving Frameworks” AND [Ali, page 8, sec VI-A] “ML Applications. We use popular computer vision ML models with serving requests extracted from the ImageNet22K dataset” AND [Ali, page 7, sec V] “The most time consuming task of the Profiler is measuring the service time of the ML application under different system configurations (i.e., memory size, batch size, and timeout).” AND [Ali, page 7, sec V] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression” AND [Ali, page 7, sec V] “the batched requests are processed through the ML model for inference.” AND [Ali, page 8, sec VI-B] “BATCH uses a multivariable regression model that is trained with a few configurations” AND [Ali, page 8, sec VI-C] “We evaluate model accuracy and robustness on AWS Lambda using different applications (i.e., MoBiNet, ResNet-v2, and ResNet-18)”, wherein the examiner interprets fitting the multivariable regression model to the service times measured by the Profiler to be the same as training the prediction machine learning model using a training dataset because they are both directed to fitting a predictive model to measured performance data. The examiner further interprets the service times that the Profiler measures while the ML applications (e.g., MoBiNet, ResNet-v2, and ResNet-18) process batched inference requests on AWS Lambda functions configured with different memory sizes to be the same as a set of metrics obtained by at least causing the set of computing instances to execute a set of machine learning models performing inferencing operations because they are both directed to performance measurements collected by running several inference models on several resource configurations.)
Romero, Ali, Justus, Li, Kotini, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting inference-serving configurations and using latency or service-time information associated with machine learning model execution.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 2 disclosed by Romero, Ali, Justus, Li, and Kotini to include the “multivariable regression model that is trained with a few configurations” disclosed by Ali. One would be motivated to do so to effectively generate predicted service-time or latency information for additional inference-serving configurations without requiring exhaustive profiling of every possible configuration, as suggested by Ali ([Ali, page 7, sec V] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”).
Regarding claim 5, Romero, Ali, Justus, Li, and Kotini teach The method of claim 1 (see rejection of claim 1).
Romero does not teach wherein the method further comprises: obtaining a forecast indicating a number of inferencing requests expected over a second interval of time;…and determining the batch size of the set of batch sizes to use to process the inferencing requests over the second interval of time based on the latency information.
Ali teaches:
wherein the method further comprises: obtaining a forecast indicating a number of inferencing requests expected over a second interval of time; ([Ali, page 5, sec III-A] “To determine the workload arrival process, the Profiler observes the inter-arrival times of incoming requests. BATCH uses the KPC-Toolbox [42] to fit the collected arrival trace into a Markovian Arrival Process (MAP) [43], a class of processes that can capture burstiness.”, wherein the examiner interprets the Markovian Arrival Process fitted to the collected arrival trace of incoming requests to be the same as a forecast indicating a number of inferencing requests expected over a second interval of time because they are both directed to a model of how many inference requests are expected to arrive over an upcoming period, which BATCH then uses to size its batches.)
and determining the batch size of the set of batch sizes to use to process the inferencing requests over the second interval of time based on the latency information. ([Ali, page 5, sec III-B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.” AND [Ali, page 5, sec III-C] “The Performance Optimizer is the core component of BATCH. It uses the arrival process and service time (both estimated by the profiler) to predict the time required to serve the incoming requests.”, wherein the examiner further interprets “determines the optimal batch size based on the arrival process, service times, SLO, and budget” to be the same as determining the batch size of the set of batch sizes to use to process the inferencing requests because they are both directed to selecting a batch size for processing inference requests. The examiner further interprets “service time” and “predict the time required to serve the incoming requests” to be the same as latency information because they are both directed to time-based performance information for serving inference requests.)
Romero, Ali, Justus, Li, Kotini, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting batch-size and computing-resource configurations based on workload demand, latency, and performance considerations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 1 disclosed by Romero, Ali, Justus, Li, and Kotini to include the Profiler that fits the incoming requests to a “Markovian Arrival Process” and the performance optimizer that “determines the optimal batch size based on the arrival process, service times, SLO, and budget” disclosed by Ali. One would be motivated to do so to efficiently meet latency objectives while reducing cost as the request load changes, as suggested by Ali ([Ali, page 4, sec III] “BATCH determines the best system configuration (i.e., memory size, maximum batch size, and timeout) to meet user-defined SLOs while reducing the cost of processing requests on a serverless platform.”).
Regarding claim 6, Romero, Ali, Justus, Li, and Kotini teach The method of claim 1 (see rejection of claim 1).
Romero further teaches:
wherein the method further comprises filtering the latency information to remove computing instances of the set of computing instances associated with a latency value over a threshold latency value. ([Romero, page 402, sec 4.1] “getVariant first enquires the Metadata Store and retrieves the variant with the lowest combined loading and inference latency that matches the query’s requirements (Line 5). If such a variant is found, INFaaS sends the query to the worker with the lowest utilization on the variant’s target hardware (Lines 6-7). Otherwise, since no registered or generated variant can meet the developer’s requirements, INFaaS suggests a variant that can achieve the closest target accuracy and/or latency (Line 8).”, wherein the examiner interprets retrieving “the variant with the lowest combined loading and inference latency that matches the query’s requirements”, sending “the query to the worker with the lowest utilization” on the variant’s target hardware, and suggesting a variant only when no registered or generated variant can meet the requirements to be the same as filtering the latency information to remove computing instances associated with a latency value over a threshold latency value because they are both directed to using latency requirements to select acceptable computing resources and exclude resources that do not meet the latency requirement.)
Regarding claim 7, Romero, Ali, Justus, Li, and Kotini teach The method of claim 1 (see rejection of claim 1).
Romero further teaches:
wherein the set of model parameters associated with the machine learning model and the number of floating point operations (FLOPS) are determined based at least in part on the machine learning model. ([Romero, page 400, sec 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID). Models for different prediction tasks within the same application (e.g., optical character recognition and language translation) can be registered with separate appIDs. Lines 1-2 in Figure 2 show how a developer registers two models, a ResNet50 and a MobileNet, for an application with appID=detectFaceApp. INFaaS generates multiple variants from these already trained models.” AND [Romero, page 397, sec 1] “These variants vary across many dimensions: the accuracies range from 56.6% to 82.5% (1.46×), the model loading latencies range from 590ms to 11s (18.7×), and the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets the model accepted by the register_model API, including modelName and modelBinary in serialized format, to be the same as the set of model parameters associated with the machine learning model because they are both directed to information identifying and representing the trained machine learning model. The examiner further interprets INFaaS generating multiple variants from already trained models, with the variants having computational requirements ranging from 0.48 to 24 GFLOPS, to be the same as determining the number of floating point operations (FLOPS) based at least in part on the machine learning model because they are both directed to obtaining floating point computational requirements from model-specific variants derived from the machine learning model.)
Regarding claim 10, Romero, Ali, Justus, and Li teach The medium of claim 9 (see rejection of claim 9).
Romero further teaches:
wherein the set of weighted sums is determined based on a first weight value associated with latency of the set of computing instances ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j , this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant” AND [Romero, page 403, sec 4.2.1] “Large values of λ place more weight on minimizing loading latency to meet SLOs when the query load is unpredictable or spiky. Small values of λ place more weight on minimizing the hardware cost when the query load is more stable.”, wherein the examiner interprets λ, which places more weight on the loading latency, to be the same as a first weight value associated with latency of the set of computing instances because they are both directed to a factor that scales the latency contribution to the computed value)
Romero does not teach and a second weight value associated with cost of the set of computing instances.
Kotini teaches and a second weight value associated with cost of the set of computing instances. ([Kotini, page 11] “Deployment goal importance Select the dropdowns below to adjust deployment goal importance. Cost Medium importance Latency High importance Throughput High importance” AND [Kotini, page 11] “All runs EC2 Instance Est. cost/hour Cost/1M inferences MaximumInvocations ModelLatency”, wherein the examiner interprets the separately selectable Cost importance level to be the same as a second weight value associated with cost of the set of computing instances because they are both directed to a user-set weighting of cost when ranking instance configurations.)
Romero, Ali, Justus, Li, Kotini, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference based on latency and cost.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model-variant selection and autoscaling technique disclosed by Romero to include the separately adjustable “Cost” and “Latency” deployment goal importance disclosed by Kotini. One would be motivated to do so to efficiently let a user balance cost against latency when choosing an instance configuration, as suggested by Kotini ([Kotini, page 8] “You can use these metrics to analyze and find the best configurations that satisfy latency, concurrency, and cost requirements of your ML application.”).
Claims 8-9, 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Romero in view of Ali in view of Justus further in view of Li.
Regarding claim 8, Romero teaches:
A non-transitory computer-readable medium storing executable instructions embodied thereon, which, when executed by a processing device, cause the processing device to perform operations comprising: ([Romero, page 404, sec 5] “We implemented INFaaS in about 20K lines of C++ code” AND [Romero, page 404, sec 5] “INFaaS’ API and communication logic between controller and workers are implemented using gRPC in C++ [2].”, wherein the examiner interprets the C++ code implementing INFaaS on the controller and worker machines to be the same as executable instructions stored on a non-transitory computer-readable medium and executed by a processing device because they are both directed to stored software run by computing hardware.)
the first machine learning model taking as inputs a set of parameters of a second machine learning model and a number of floating point operations (FLOPs) associated with the second machine learning model ([Romero, page 400, sec 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID).” AND [Romero, page 397, sec 1] “and the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets the model (modelBinary) in serialized format to be the same as the set of parameters of a second machine learning model because they are both directed to data defining a trained machine learning model; and further interprets the computational requirements of 0.48 to 24 GFLOPS to be the same as the number of floating point operations (FLOPs) associated with the second machine learning model because they are both directed to the floating point workload of executing a model for inference.)
determining a computing instance of the set of computing instances and a batch size of the set of batch sizes to execute an inferencing service using the second machine learning model based on the latency information; and ([Romero, page 402, sec 4.1] “When a query arrives, INFaaS’ Dispatcher invokes the getVariant method of model-variant selection policy to choose a variant (Algorithm 1). For this case, the input to getVariant is the query’s requirements, and the output is the variant and worker to serve the query…getVariant first enquires the Metadata Store and retrieves the variant with the lowest combined loading and inference latency that matches the query’s requirements” AND [Romero, page 398, sec 2.1] “(d) hyperparameters (e.g., optimizing for batch size of 1, 4, 8, or 16), and (e) hardware platforms (e.g., Haswell or Skylake CPUs, V100 or T4 GPUs, FPGA, and accelerators, such as Inferentia [18], TPU [49], Catapult [32], NPU [7])”, wherein the examiner interprets outputting the variant, which is defined by its batch size and hardware platform, and the worker to serve the query to be the same as determining a computing instance and a batch size to execute an inferencing service using the second machine learning model because they are both directed to choosing the hardware and batch configuration that serves inference queries; and further interprets retrieving the variant with the lowest combined loading and inference latency to be the same as determining based on the latency information because they are both directed to selection driven by latency values.)
causing the computing instance to execute the second machine learning model ([Romero, page 400, sec 3.1] “The Controller selects a model-variant and dispatches the inference query to a Worker machine. Workers further send inference queries to the appropriate Hardware Executors according to the selected model-variant, and reply with inference results to applications.” AND [Romero, page 400, sec 3.2] “Worker machines execute inference queries assigned by the controller. Hardware-specific Executor daemons manage the deployment and execution of model-variants.”, wherein the examiner interprets dispatching the inference query to the worker whose hardware executor executes the selected model-variant to be the same as causing the computing instance to execute the second machine learning model because they are both directed to running the model on the selected hardware resource.)
Romero does not teach causing a first machine learning model to predict latency information associated with a set of computing instances executing an inferencing request associated with a set of batch sizes,…wherein the first machine learning model has been trained, based on latency information collected for different combinations of machine learning models, computing instance configurations, and batch sizes, to predict latency values for combinations of the set of computing instances and the set of batch sizes using the set of parameters…and the number of FLOPs of the second machine learning model;…and process inferencing requests including a number of data objects corresponding to the batch size.
Ali teaches causing a first machine learning model to predict latency information associated with a set of computing instances executing an inferencing request associated with a set of batch sizes, ([Ali, page 5, sec III-A] “the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory. BATCH derives the batch service time model using multivariable polynomial regression [44]” AND [Ali, page 7, sec V] “To reduce the profiling time, the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression.” AND [Ali, page 8, sec VI-B] “To determine the mean of the service time distribution with as few experiments as possible for the various memory/batch size configurations, BATCH uses a multivariable regression model that is trained with a few configurations”, wherein the examiner interprets the multivariable regression model that is trained with a few configurations to be the same as the first machine learning model because they are both directed to a model trained on measured data to estimate unmeasured performance values, and the amounts of allocated memory of the memory/batch size configurations to be the same as the set of computing instances because they are both directed to compute resource configurations that execute the inference; and further interprets the estimated batch service time for batches of different sizes to be the same as latency information associated with a set of batch sizes because they are both directed to the time needed to process batches of inference requests of given sizes.)
and process inferencing requests including a number of data objects corresponding to the batch size. ([Ali, page 7, sec V] “As soon as the function is invoked, the list of requests are transformed into a batch by the serving function and the batched requests are processed through the ML model for inference.” AND [Ali, page 5, sec III-B] “Since serverless platforms do not automatically allow processing multiple requests in a single batch, BATCH implements a Buffer to batch together requests for serving.”, wherein the examiner interprets the list of requests transformed into a batch and processed through the ML model to be the same as processing inferencing requests including a number of data objects corresponding to the batch size because they are both directed to grouping inference inputs into a batch and running them through the model.)
Romero and Ali do not teach wherein the first machine learning model has been trained, based on latency information collected for different combinations of machine learning models, computing instance configurations, and batch sizes, to predict latency values for combinations of the set of computing instances and the set of batch sizes using the set of parameters…and the number of FLOPs of the second machine learning model;.
Justus teaches wherein the first machine learning model has been trained, based on latency information collected for different combinations of machine learning models, computing instance configurations, and batch sizes, to predict latency values for combinations of the set of computing instances and the set of batch sizes using the set of parameters ([Justus, page 4, sec V-A] “We construct a random selection of these atomic operations embedded within the simplest network we can produce. Each of these atomic operations is then executed either using forward or forwards and backwards passes and the execution times are recorded from multiple executions. The feature set for the atomic operation along with the execution timings are then used to train a fully connected feed forward network. This network can then be used for predicting the execution time for a new operation based just on the feature set.” AND [Justus, page 3, sec IV-A] “These relate to those features of a particular layer within the neural network and in particular to the hyperparameters related to that layer.” AND [Justus, page 3, sec IV-A] “Batch size representing the number of training samples which are processed together as part of the same batch.” AND [Justus, page 4, sec IV-C] “Hardware features are those describing both the GPU card(s) used and the system these cards are housed within” AND [Justus, page 6, sec VII] “We then test our deep learning predictor when it has been trained on data from multiple GPU types. In this second case we use the trained model to predict execution times for a GPU type not seen as part of the training process.”, wherein the examiner interprets the execution timings recorded for the randomly selected networks, having varying layer hyperparameters and batch sizes, on the multiple GPU types described by hardware features to be the same as latency information collected for different combinations of machine learning models, computing instance configurations, and batch sizes because they are both directed to measured execution times across varied models, hardware, and batch sizes; and further interprets predicting the execution time for a new operation based just on its feature set to be the same as predicting latency values for combinations of the set of computing instances and the set of batch sizes using the set of parameters because they are both directed to estimating execution time from model, hardware, and batch-size features without running the model.)
Romero, Ali, and Justus do not teach and the number of FLOPs of the second machine learning model;.
Li teaches and the number of FLOPs of the second machine learning model; ([Li, page 106, sec 4.3.2] “We built a synthetic dataset including 1000 neural architectures sampled from this NAS space. For each neural architecture, we collected training measurements on 6 mobile platforms (Table 1), for a total of 90 scenarios” AND [Li, page 106, Table 3] “Input height (width), input channel, output height (width), stride, kernel height (width), filters, input size, output size, kernel size, FLOPs”, wherein the examiner interprets the FLOPs feature of the operations of a neural architecture whose latency is predicted to be the same as the number of FLOPs of the second machine learning model used to predict latency values because they are both directed to the floating point operation count of the model being evaluated as a predictor input.)
Romero, Ali, Justus, Li, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model-variant selection and autoscaling technique disclosed by Romero to include the “multivariable regression model that is trained with a few configurations” and the serving function in which “the list of requests are transformed into a batch” disclosed by Ali. One would be motivated to do so to efficiently obtain batch service times for the various configurations without exhaustively profiling every configuration, as suggested by Ali ([Ali, page 8, sec VI-B] “Results show that BATCH can reduce the profiling time by more than 97% compared to exhaustively profiling all system configurations.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model-variant selection and autoscaling technique disclosed by Romero to include the execution timings and features used to “train a fully connected feed forward network” disclosed by Justus. One would be motivated to do so to effectively predict execution times for previously unseen models and hardware, as suggested by Justus ([Justus, page 3, sec III] “Our work aims to fill this gap by providing a prediction framework that is capable of dealing with these non-linearities, providing accurate results while also generalising to previously unseen neural network models or hardware.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model-variant selection and autoscaling technique disclosed by Romero to include the “FLOPs” features disclosed by Li. One would be motivated to do so to accurately predict end-to-end inference latency, as suggested by Li ([Li, Abstract] “achieves high accuracy in end-to-end latency predictions”).
Regarding claim 9, Romero, Ali, Justus, and Li teach The medium of claim 8 (see rejection of claim 8).
Romero further teaches wherein the processing device further performs operations comprising: generating a set of weighted sums associated with the set of computing instances and the set of batch sizes based on the latency information; and wherein determining the computing instance is further determined based on the set of weighted sums. ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j, this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant: Cost(δi j) = Ci j(δi j +λT load i j max(δi j,0))” AND [Romero, page 403, sec 4.2.2] “Finally, the model-variant selection policy computes the cost function of our ILP, by using the hardware cost ($/s) and the variant loading latency to decide whether to replicate the running variant, or upgrade to a variant that supports higher throughput.”, wherein the examiner interprets the cost for an action being the sum of the hardware cost and the loading latency of the variant to be the same as generating a set of weighted sums associated with the set of computing instances and the set of batch sizes based on the latency information because they are both directed to computing summed selection values that include latency information for inference-serving resource options. The examiner further interprets using the hardware cost and the variant loading latency to decide whether to replicate the running variant or upgrade to a variant that supports higher throughput to be the same as determining the computing instance based on the set of weighted sums because they are both directed to selecting a computing resource configuration based on computed cost-latency values.)
Regarding claim 11, Romero, Ali, Justus, and Li teach The medium of claim 8 (see rejection of claim 8).
Romero does not teach wherein the latency information indicates an amount of time computing instances of the set of computing instances take to process an inferencing request of batch sizes of the set of batch sizes using the second machine learning model.
Ali teaches wherein the latency information indicates an amount of time computing instances of the set of computing instances take to process an inferencing request of batch sizes of the set of batch sizes using the second machine learning model. ([Ali, page 5, sec III-A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.” AND [Ali, page 6, sec IV] “where Sk is the the service time of a batch of size k.”, wherein the examiner interprets “the time required to process batches of different sizes” and “the service time of a batch of size k” to be the same as latency information indicating an amount of time computing instances take to process an inferencing request of batch sizes of the set of batch sizes because they are both directed to time-based performance information for processing batched inference requests. The examiner further interprets the serverless platform and allocated memory to be the same as computing instances of the set of computing instances because they are both directed to computing resources used to execute the inference processing. The examiner further interprets processing batches of different sizes on the serverless platform using an ML serving application to be the same as processing an inferencing request of batch sizes using the second machine learning model because they are both directed to executing batched inference requests using a deployed machine learning model.)
Romero, Ali, Justus, Li, and the instant application are analogous art because they are all directed to machine learning inference serving, including determining latency or service-time information for batched inference requests executed using computing resources.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero, Ali, Justus, and Li to include “the Profiler measures the time required to process batches of different sizes” disclosed by Ali. One would be motivated to do so to effectively evaluate processing time for different batch-size configurations so that inference-serving configurations can be selected based on latency information, as suggested by Ali ([Ali, page 5, sec III-A] “the Profiler measures the time required to process batches of different sizes”).
Regarding claim 12, Romero, Ali, Justus, and Li teach The medium of claim 8 (see rejection of claim 8).
Romero does not teach wherein the processing device further performs operations comprising: obtaining a forecast predicting a number of inferencing requests over an interval of time; and wherein determining the batch size is further determined based on the forecast.
Ali teaches wherein the processing device further performs operations comprising: obtaining a forecast predicting a number of inferencing requests over an interval of time; and wherein determining the batch size is further determined based on the forecast. ([Ali, page 5, sec III-A] “To determine the workload arrival process, the Profiler observes the inter-arrival times of incoming requests. BATCH uses the KPC-Toolbox [42] to fit the collected arrival trace into a Markovian Arrival Process (MAP) [43], a class of processes that can capture burstiness.” AND [Ali, page 5, sec III-B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”, wherein the examiner interprets determining the workload arrival process using observed inter-arrival times of incoming requests and fitting the collected arrival trace into a Markovian Arrival Process to be the same as obtaining a forecast predicting a number of inferencing requests over an interval of time because they are both directed to modeling the timing/rate of incoming inference requests over time. The examiner further interprets determining the optimal batch size based on the arrival process to be the same as determining the batch size based on the forecast because they are both directed to selecting a batch size using modeled incoming request information.)
Romero, Ali, Justus, Li, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting batch-size and inference-serving configurations based on modeled incoming request information and latency-based performance considerations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero, Ali, Justus, and Li to include the “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget” disclosed by Ali. One would be motivated to do so to efficiently select a batch size using modeled inference-request arrivals so that the selected batch size satisfies latency and performance goals under bursty or changing workloads, as suggested by Ali ([Ali, page 5, sec III-B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”).
Regarding claim 13, Romero, Ali, Justus, and Li teach The medium of claim 8 (see rejection of claim 8).
Romero further teaches wherein the processing device further performs operations comprising filtering computing instances of the set of computing instances based on a latency threshold. ([Romero, page 402, sec 4.1] “getVariant first enquires the Metadata Store and retrieves the variant with the lowest combined loading and inference latency that matches the query’s requirements (Line 5). If such a variant is found, INFaaS sends the query to the worker with the lowest utilization on the variant’s target hardware (Lines 6-7). Otherwise, since no registered or generated variant can meet the developer’s requirements, INFaaS suggests a variant that can achieve the closest target accuracy and/or latency (Line 8).”, wherein the examiner interprets retrieving the variant with the lowest combined loading and inference latency that matches the query’s requirements and sending the query to the worker with the lowest utilization on the variant’s target hardware to be the same as filtering computing instances of the set of computing instances based on a latency threshold because they are both directed to selecting acceptable inference-serving computing resources based on whether latency satisfies a required latency criterion.)
Regarding claim 14, Romero, Ali, Justus, and Li teach The medium of claim 8 (see rejection of claim 8).
Romero does not teach wherein the first machine learning model is a regression model.
Ali teaches wherein the first machine learning model is a regression model. ([Ali, page 8, sec VI-B] “BATCH uses a multivariable regression model that is trained with a few configurations” AND [Ali, page 7, sec V] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”, wherein the examiner interprets the multivariable regression model and estimating service times through regression to be the same as the first machine learning model being a regression model because they are both directed to using a regression-based predictive model to estimate latency/service-time information for inference-serving configurations.)
Romero, Ali, Justus, Li, and the instant application are analogous art because they are all directed to machine learning inference serving, including predicting latency or service-time information for selecting inference-serving configurations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero, Ali, Justus, and Li to include the “multivariable regression model that is trained with a few configurations” disclosed by Ali. One would be motivated to do so to effectively estimate service-time or latency information for additional inference-serving configurations without requiring exhaustive profiling of every possible configuration, as suggested by Ali ([Ali, page 7, sec V] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”).
Regarding claim 15, Romero, Ali, Justus, and Li teach The medium of claim 8 (see rejection of claim 8).
Justus further teaches wherein the processing device further performs operations comprising training the first machine learning model based on a training dataset including latency obtained from the set of computing instances executing inferencing requests using a set of machine learning models, parameters associated with the set of machine learning models, ([Justus, page 4, sec V-A] “We construct a random selection of these atomic operations embedded within the simplest network we can produce. Each of these atomic operations is then executed either using forward or forwards and backwards passes and the execution times are recorded from multiple executions. The feature set for the atomic operation along with the execution timings are then used to train a fully connected feed forward network.” AND [Justus, page 6, sec VI] “50% of all experiments were performed as pure forward passes without any backpropagation.” AND [Justus, page 3, sec IV-A] “These relate to those features of a particular layer within the neural network and in particular to the hyperparameters related to that layer.” AND [Justus, page 6, sec VII] “We then test our deep learning predictor when it has been trained on data from multiple GPU types. In this second case we use the trained model to predict execution times for a GPU type not seen as part of the training process.”, wherein the examiner interprets training the fully connected feed forward network on execution timings recorded from forward passes of the randomly selected networks on multiple GPU types to be the same as training the first machine learning model based on a training dataset including latency obtained from the set of computing instances executing inferencing requests using a set of machine learning models because they are both directed to fitting a predictor to measured execution times of multiple models on multiple hardware resources. The examiner further interprets the layer hyperparameter features to be the same as parameters associated with the set of machine learning models because they are both directed to values defining the architecture of each model.)
Li further teaches and FLOPS associated with the set of machine learning models. ([Li, page 106, sec 4.3.2] “We built a synthetic dataset including 1000 neural architectures sampled from this NAS space. For each neural architecture, we collected training measurements on 6 mobile platforms (Table 1), for a total of 90 scenarios” AND [Li, page 106, Table 3] “Input height (width), input channel, output height (width), stride, kernel height (width), filters, input size, output size, kernel size, FLOPs”, wherein the examiner interprets the FLOPs feature computed for the operations of each of the 1000 neural architectures to be the same as FLOPS associated with the set of machine learning models because they are both directed to floating point operation counts of multiple models used as inputs for training a latency predictor)
Romero, Ali, Justus, Li, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero, Ali, Justus, and Li to include the execution timings and features used to “train a fully connected feed forward network” disclosed by Justus. One would be motivated to do so to effectively predict execution times for previously unseen models and hardware, as suggested by Justus ([Justus, page 3, sec III] “Our work aims to fill this gap by providing a prediction framework that is capable of dealing with these non-linearities, providing accurate results while also generalising to previously unseen neural network models or hardware.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the “FLOPs” features disclosed by Li. One would be motivated to do so to accurately predict end-to-end inference latency, as suggested by Li ([Li, Abstract] “achieves high accuracy in end-to-end latency predictions”).
Claims 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ali in view of Justus further in view of Li.
Regarding claim 16, Ali teaches:
A system comprising: a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising: ([Ali, page 8, Fig. 9] “CPU and memory usage over three hours of an AWS t2.nano instance hosting BATCH components.” AND [Ali, page 7, sec V] “Since this is an offline phase and the profiler requires little computational power, it can be collocated with the Buffer and Performance optimizer.”, wherein the examiner interprets the AWS t2.nano instance whose CPU and memory host the BATCH components to be the same as a system comprising a memory component and a processing device coupled to the memory component because they are both directed to computing hardware that executes the serving operations.)
obtaining a training dataset including latency information obtained from a plurality of computing instances executing a plurality of inferencing requests of a plurality of batch sizes ([Ali, page 7, sec V] “The most time consuming task of the Profiler is measuring the service time of the ML application under different system configurations (i.e., memory size, batch size, and timeout).” AND [Ali, page 5, sec III-A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.”, wherein the examiner interprets the service times measured under different memory size and batch size configurations, which are used to fit the regression model, to be the same as a training dataset including latency information obtained from a plurality of computing instances executing a plurality of inferencing requests of a plurality of batch sizes because they are both directed to measured inference times across resource and batch configurations used to fit a predictor.)
providing the prediction machine learning model to a latency tool; and ([Ali, page 4, sec III] “and uses simple regression to capture the relationship between system configuration and request service times…Using as inputs the fitted arrival process, request service times for different system configurations, the monetary budget, and the user SLO, BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies” AND [Ali, page 7, sec V] “Since this is an offline phase and the profiler requires little computational power, it can be collocated with the Buffer and Performance optimizer.”, wherein the examiner interprets supplying the regression-based service times to the Performance Optimizer, which predicts the distribution of latencies, to be the same as providing the prediction machine learning model to a latency tool because they are both directed to making the trained predictor available to a component that evaluates latency.)
causing the latency tool to determine a computing instance and a batch size for processing a set of inferencing requests using the machine learning model based at least in part on the predicted latency information generated by the prediction machine learning model, ([Ali, page 4, sec III] “BATCH determines the best system configuration (i.e., memory size, maximum batch size, and timeout) to meet user-defined SLOs while reducing the cost of processing requests on a serverless platform.” AND [Ali, page 5, sec III-C] “It uses the arrival process and service time (both estimated by the profiler) to predict the time required to serve the incoming requests.”, wherein the examiner interprets the Performance Optimizer determining the memory size and maximum batch size to be the same as the latency tool determining a computing instance and a batch size because they are both directed to selecting the resource configuration and batch size used to serve requests, and that the the service time estimated by the profiler to be the same as the predicted latency information generated by the prediction machine learning model because they are both directed to estimated processing times output by the trained predictor.)
Ali does not teach using a plurality of machine learning models, wherein the training dataset associates the latency information with computing instance configuration information for the plurality of computing instances, batch sizes of the plurality of inferencing requests, model parameters associated with the plurality of machine learning models,…and numbers of FLOPs associated with the plurality of machine learning models;…training a prediction machine learning model to generate, for a machine learning model and based on model parameters…and a number of FLOPs associated with the machine learning model,…predicted latency information for a set of computing instances and a set of batch sizes using the training dataset;…the prediction machine learning model taking as an input information associated with the machine learning model.
Justus teaches using a plurality of machine learning models, wherein the training dataset associates the latency information with computing instance configuration information for the plurality of computing instances, batch sizes of the plurality of inferencing requests, model parameters associated with the plurality of machine learning models, ([Justus, page 4, sec V-A] “We construct a random selection of these atomic operations embedded within the simplest network we can produce. Each of these atomic operations is then executed either using forward or forwards and backwards passes and the execution times are recorded from multiple executions. The feature set for the atomic operation along with the execution timings are then used to train a fully connected feed forward network.” AND [Justus, page 3, sec IV-A] “These relate to those features of a particular layer within the neural network and in particular to the hyperparameters related to that layer.” AND [Justus, page 3, sec IV-A] “Batch size representing the number of training samples which are processed together as part of the same batch.” AND [Justus, page 4, sec IV-C] “Hardware features are those describing both the GPU card(s) used and the system these cards are housed within” AND [Justus, page 6, sec VII] “We then test our deep learning predictor when it has been trained on data from multiple GPU types. In this second case we use the trained model to predict execution times for a GPU type not seen as part of the training process.”, wherein the examiner interprets the training data pairing the execution timings with the hardware features of the multiple GPU types, the batch size, and the layer hyperparameters of the randomly selected networks to be the same as a training dataset that associates the latency information with computing instance configuration information, batch sizes, and model parameters for a plurality of machine learning models because they are both directed to measured execution times keyed to hardware, batch-size, and model features.)
Justus teaches:
training a prediction machine learning model to generate, for a machine learning model and based on model parameters…predicted latency information for a set of computing instances and a set of batch sizes using the training dataset; ([Justus, page 4, sec V-A] “We construct a random selection of these atomic operations embedded within the simplest network we can produce. Each of these atomic operations is then executed either using forward or forwards and backwards passes and the execution times are recorded from multiple executions. The feature set for the atomic operation along with the execution timings are then used to train a fully connected feed forward network. This network can then be used for predicting the execution time for a new operation based just on the feature set.” AND [Justus, page 6, sec VII] “We then test our deep learning predictor when it has been trained on data from multiple GPU types. In this second case we use the trained model to predict execution times for a GPU type not seen as part of the training process.”, wherein the examiner interprets training the fully connected feed forward network to predict the execution time for a new operation, on seen or unseen GPU types, from its layer hyperparameters, batch size, and hardware features to be the same as training a prediction machine learning model to generate, for a machine learning model and based on model parameters, predicted latency information for a set of computing instances and a set of batch sizes using the training dataset because they are both directed to a learned predictor of execution time across hardware and batch-size configurations.)
the prediction machine learning model taking as an input information associated with the machine learning model. ([Justus, page 3, sec IV-A] “These relate to those features of a particular layer within the neural network and in particular to the hyperparameters related to that layer.” AND [Justus, page 4, sec V-A] “This network can then be used for predicting the execution time for a new operation based just on the feature set.”, wherein the examiner interprets the layer features input to the trained network to be the same as information associated with the machine learning model taken as an input because they are both directed to descriptive data about the model being evaluated)
Ali and Justus do not teach and numbers of FLOPs associated with the plurality of machine learning models;…and a number of FLOPs associated with the machine learning model,.
Li teaches and numbers of FLOPs associated with the plurality of machine learning models;…and a number of FLOPs associated with the machine learning model, ([Li, page 106, sec 4.3.2] “We built a synthetic dataset including 1000 neural architectures sampled from this NAS space. For each neural architecture, we collected training measurements on 6 mobile platforms (Table 1), for a total of 90 scenarios” AND [Li, page 106, Table 3] “Input height (width), input channel, output height (width), stride, kernel height (width), filters, input size, output size, kernel size, FLOPs”, wherein the examiner interprets the FLOPs feature computed for the operations of the neural architectures to be the same as the numbers of FLOPs associated with the plurality of machine learning models and the number of FLOPs associated with the machine learning model because they are both directed to floating point operation counts of models used as latency-predictor inputs.)
Ali, Justus, Li, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the regression-based profiling and performance optimization technique disclosed by Ali to include the execution timings and features used to “train a fully connected feed forward network” disclosed by Justus. One would be motivated to do so to effectively predict execution times for previously unseen models and hardware, as suggested by Justus ([Justus, page 3, sec III] “Our work aims to fill this gap by providing a prediction framework that is capable of dealing with these non-linearities, providing accurate results while also generalising to previously unseen neural network models or hardware.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the “FLOPs” features disclosed by Li. One would be motivated to do so to accurately predict end-to-end inference latency, as suggested by Li ([Li, Abstract] “achieves high accuracy in end-to-end latency predictions”).
Regarding claim 18, Ali, Justus, and Li teach The system of claim 16 (see rejection of claim 16).
Ali further teaches wherein the latency information indicates an amount of time a first computing instance of the plurality of computing instances takes to process an inferencing request of a first batch size of the plurality of batch sizes. ([Ali, page 5, sec III-A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.” AND [Ali, page 6, sec IV] “where Sk is the the service time of a batch of size k.”, wherein the examiner interprets “the time required to process batches of different sizes” and “the service time of a batch of size k” to be the same as latency information indicating an amount of time to process an inferencing request of a first batch size of the plurality of batch sizes because they are both directed to the time required to process batched inference requests having a particular batch size. The examiner further interprets the serverless platform and allocated memory to be the same as a first computing instance of the plurality of computing instances because they are both directed to a computing resource used to execute inference processing.)
Regarding claim 19, Ali, Justus, and Li teach The system of claim 16 (see rejection of claim 16).
Ali further teaches wherein the prediction machine learning model is a…regression model. ([Ali, page 8, section 6-B] “To determine the mean of the service time distribution with as few experiments as possible for the various memory/batch size configurations, BATCH uses a multivariable regression model that is trained with a few configurations (in our experiments, less than 3%). Results show that BATCH can reduce the profiling time by more than 97% compared to exhaustively profiling all system configurations. The accuracy of the regression model is validated against configurations that are not used for training. The mean absolute percentage error is always less than 2%.”, wherein the examiner interprets the multivariable regression model trained with a few configurations to be the same as the prediction machine learning model because they are both directed to a trained predictive model used to determine latency or service-time information for inference-serving configurations.).
Li further teaches random forest ([Li, page 105, sec 4.2] “Random Forests (RF). An RF model includes multiple decision trees to reduce the overfitting of a single decision tree.”, wherein the examiner interprets the Random Forests latency predictor to be the same as a random forest regression model because they are both directed to an ensemble of decision trees that regresses a latency value.)
Ali, Justus, Li, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of claim 16 disclosed by Ali, Justus, and Li to include the multivariable regression model trained with a few configurations disclosed by Ali. One would be motivated to do so to effectively to use the regression learning model to be trained on configurations, for which can include predictive data, to determine latency or other information as suggested by Ali ([Ali, page 8, section 6-B] “To determine the mean of the service time distribution with as few experiments as possible for the various memory/batch size configurations, BATCH uses a multivariable regression model that is trained with a few configurations (in our experiments, less than 3%). Results show that BATCH can reduce the profiling time by more than 97% compared to exhaustively profiling all system configurations. The accuracy of the regression model is validated against configurations that are not used for training. The mean absolute percentage error is always less than 2%.”)
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the “Random Forests (RF)” latency predictor disclosed by Li. One would be motivated to do so to effectively reduce prediction overfitting, as suggested by Li ([Li, page 105, sec 4.2] “An RF model includes multiple decision trees to reduce the overfitting of a single decision tree.”).
Regarding claim 20, Ali, Justus, and Li teach The system of claim 16 (see rejection of claim 16).
Ali further teaches wherein causing the latency tool to determine the computing instance and the batch size further comprises determining the computing instance and the batch size based on a forecast indicating a number of inferencing requests over an interval of time. ([Ali, page 5, sec III-A] “To determine the workload arrival process, the Profiler observes the inter-arrival times of incoming requests. BATCH uses the KPC-Toolbox [42] to fit the collected arrival trace into a Markovian Arrival Process (MAP) [43], a class of processes that can capture burstiness.” AND [Ali, page 5, sec III-B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”, wherein the examiner interprets determining the workload arrival process using observed inter-arrival times of incoming requests and fitting the collected arrival trace into a Markovian Arrival Process to be the same as a forecast indicating a number of inferencing requests over an interval of time because they are both directed to modeling incoming inference-request activity over time. The examiner further interprets determining the optimal batch size based on the arrival process to be the same as determining the batch size based on the forecast because they are both directed to selecting a batch size using modeled incoming request information.)
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Ali in view of Justus in view of Li further in view of Romero.
Regarding claim 17, Ali, Justus, and Li teach The system of claim 16 (see rejection of claim 16).
Ali does not teach wherein the information associated with the machine learning model includes at least one of: a number of floating point operations (FLOPs), a number of layers, a number of activations, and a number of parameters.
Romero teaches wherein the information associated with the machine learning model includes at least one of: a number of floating point operations (FLOPs), a number of layers, a number of activations, and a number of parameters. ([Romero, page 400, sec 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID).” AND [Romero, page 397, sec 1] “the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets modelName and modelBinary in serialized format to be the same as information associated with the machine learning model including a number of parameters because they are both directed to information identifying and representing the trained machine learning model. The examiner further interprets computational requirements ranging from 0.48 to 24 GFLOPS to be the same as information associated with the machine learning model including a number of floating point operations because they are both directed to floating point computational requirements associated with executing inference using the machine learning model.)
Ali, Justus, Li, Romero, and the instant application are analogous art because they are all directed to estimating the execution latency of machine learning models on computing resources and selecting resource configurations for machine learning inference.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of claim 16 disclosed by Ali, Justus, and Li to include the registered model information and the model “computational requirements” in “GFLOPS” disclosed by Romero. One would be motivated to do so to effectively account for the wide variation in computational requirements across models when predicting latency and selecting an inference-serving configuration, as suggested by Romero ([Romero, Abstract] “However, given the diversity across variants and hardware platforms in the cloud, a lack of understanding of the trade-off space can incur significant costs to developers.” AND [Romero, page 397, sec 1] “Being user-facing, inference serving requires cost-effective systems that render predictions with latency constraints while handling unpredictable and bursty request arrivals.”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
