DETAILED ACTION
This action is responsive to the application filed on 08/02/2023. Claims 1-20 are pending and have been examined. This action is Non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition
of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
Step 2A Prong 1:
“and a number of floating point operations (FLOPS) associated with performing inferencing operations” -- The limitation is directed to a number of floating point operations associated with performing the inferencing operations. The limitation is directed to mathematical operation/calculation/relationship, and thus the limitation is directed to math.
“determining a set of computing instances capable of executing the machine learning model and a set of batch sizes associated with inferencing requests; -- The limitation is directed to determining instances capable of executing the model/batch sizes associated with the interfering requests. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
“determining a set of weight sum values associated with the set of computing instances based on the latency information; -- This limitation is directed to calculating weighted score values based on latency information. Determining weight sum values is a mathematical calculation used to evaluate computing instances, and thus the limitation is directed to math.
Step 2A Prong 2 and Step 2B:
“A method comprising: obtaining an input to a prediction machine learning model, the input indicating a set of model parameters associated with a machine learning model...causing the prediction machine learning model to output latency information for the set of computing instances and the set of batch sizes, the latency information indicating an interval of time to process an inferencing request of a batch size of the set of batch sizes by a computing instance of the set of computing instances;” -- The limitation recites obtaining an input for a prediction ML model, where the input indicates a set of model parameters associated with the model and then causing the model to output latent information for the computing instances and batch size set, where the latency information indicates a time interval to process gathered data and information. The limitation is directed to an insignificant, extra-solution activity, that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of obtaining data and inputting/outputting related data over a network/computerized system is a well-understood, routine, and conventional activity (WURC) that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“and causing an indication of the set of weight sum values associated with the set of computing instances to be displayed in a user interface.” -- The limitation recites displaying a set of weight sum values with the set of computing instances to be displayed using an UI. The limitation amounts to no more than mere instructions to apply onto a computer, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 1 is non-patent eligible. Claim 16 is analogous to claim 1 for purposes of the 35 U.S.C. 101 analysis because both claims are directed to using a prediction machine learning model to determine latency information for computing instances and batch sizes associated with machine learning inferencing. Although claim 16 is drafted as a system and further recites training the prediction model and providing it to a latency tool, these limitations merely further describe using generic computer components to perform the same mathematical prediction and resource-selection process. Therefore, claim 16 does not materially alter the eligibility analysis set forth for claim 1, and the same can apply.
Regarding claim 2,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein the method further comprises selecting a first computing instance of the set of computing instances...,where based at least in part on a first weight sum value of the set of weight sum values.”-- The limitation is directed to selecting a first computing instance of the set of instances which is based on a part of the first weight sum value set. The limitation is directed to the use of mathematical operations/calculation/relationships, and thus is directed to math.
Step 2A Prong 2 and Step 2B:
“to execute the machine learning model”-- The limitation recites executing the ML model to complete the selecting process recited earlier in the claim. The limitation amounts to no more than mere instructions to apply onto a computer, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 2 is non-patent eligible.
Regarding claim 3,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 2, wherein the first weight sum value is greater than at least one other weight sum value of the set of weight sum values.” -- The limitation recites that the first weight sum value introduced in earlier claims will further be greater than at least one other weight sum values of the weight sum value sets. The limitation amounts to merely further limiting to a field of use/environment, and thus it cannot be integrated to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 3 is non-patent eligible.
Regarding claim 4,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 2, wherein the method further comprises training the prediction machine learning model using a training dataset including a set of metrics obtained by at least causing the set of computing instances to execute a set of machine learning models performing inferencing operations.” -- The limitation recites that the training of the prediction ML model will use a training data that includes metrics obtained by causing the computing instances to execute the ML models performing the inferencing operations. The limitation amounts to no more than mere instructions to apply onto a computer, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 4 is non-patent eligible.
Regarding claim 5,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein the method further comprises :obtaining a forecast indicating a number of inferencing requests expected over a second interval of time; and determining the batch size of the set of batch sizes to use to process the inferencing requests over the second interval of time based on the latency information.” -- The limitation is directed to obtaining a forecast of the amount of inference requests over an interval of time, and determining the batch size of the set of batch sizes to use it for processing inference requests over an interval of time based on collected information. The limitation is a process that can be performed in the human mind using evaluation, observation, and judgement, and thus the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 5 is non-patent eligible.
Regarding claim 6,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, wherein the method further comprises filtering the latency information to remove computing instances of the set of computing instances associated with a latency value over a threshold latency value.” -- The limitation recites filtering the latency information by removing computing instances associated with a latency value over a threshold latency value. The limitation amounts to no more than mere further limiting to a field of use/environment, and thus it cannot be integrated to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 6 is non-patent eligible.
Regarding claim 7,
Step 1: The claim is directed to method, which is considered to be a process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, wherein the set of model parameters associated with the machine learning model and the number of floating point operations (FLOPS) are determined based at least in part on the machine learning model.” -- The limitation recites that the model parameters for the ML model and the FLOPS are partially determined by ML model. The limitation amounts to no more than mere instructions to apply onto a computer, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 7 is non-patent eligible.
Regarding claim 8,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
Step 2A Prong 1;
“determining a computing instance of the set of computing instances and a batch size of the set of batch sizes” -- The limitation is directed to determining a computing instance and a batch size. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“A non-transitory computer-readable medium storing executable instructions embodied thereon, which, when executed by a processing device, cause the processing device to perform operations comprising: causing a first machine learning model to predict latency information associated with a set of computing instances executing an inferencing request associated with a set of batch sizes...execute an inferencing service using the second machine learning model based on the latency information...and causing the computing instance to execute the second machine learning model and process inferencing requests including a number of data objects corresponding to the batch size.” -- The limitation recites executing instructions onto a processing device which causes the device to perform operations that comprise causing a first machine learning model to predict latency information associated with a set of instances executing an inference request associated with the batch sizes, and causing the computer instance to execute (instruct to execute) the second ML model and the process requests including a number of data objects corresponding to the batch size. The limitation amounts to no more than mere instructions to apply onto a computer, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
“the first machine learning model taking as inputs a set of parameters of a second machine learning model and a number of flopping point operations (FLOPs) associated with the second machine learning model;” -- The limitation recites a machine learning model that takes inputs of parameter sets and the FLOPS associated with the second ML model. The limitation amounts to no more than mere instructions to apply onto a computer, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 8 is non-patent eligible.
Regarding claim 9,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
Step 2A Prong 1:
“The medium of claim 8, wherein the processing device further performs operations comprising: generating a set of weighted sums associated with the set of computing instances and the set of batch sizes based on the latency information; and wherein determining the computing instance is further determined based on the set of weighted sums.” -- The limitation is directed to generating a set of weighted sums associated with the set of computing instances and the batch sets based on latency information, and determining computing instances based on the set of weighted sums. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 9 is non-patent eligible.
Regarding claim 10,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
Step 2A Prong 1:
“The medium of claim 9, wherein the set of weighted sums is determined based on a first weight value associated with latency of the set of computing instances and a second weight value associated with cost of the set of computing instances.”-- The limitation is directed to determining a set of weighted sum based on values associated with latency of the computing instance set and second weight values associated with cost of the computing instance set. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 10 is non-patent eligible.
Regarding claim 11,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The medium of claim 8, wherein the latency information indicates an amount of time computing instances of the set of computing instances take to process an inferencing request of batch sizes of the set of batch sizes using the second machine learning model.” -- The limitation recites that the latency information that indicates an amount of computing instances out of the set to take to a process of an inferencing request of batch sizes out of a set using the second ML model. The limitation amounts to no more than merely limiting to a field of use/environment, and thus does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 11 is non-patent eligible.
Regarding claim 12,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
Step 2A Prong 1:
“The medium of claim 8, wherein the processing device further performs operations comprising: obtaining a forecast predicting a number of inferencing requests over an interval of time;” -- The limitation is directed to obtaining a forecast prediction of a number of inference requests over a time interval. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
“wherein determining the batch size is further determined based on the forecast.” -- The limitation is directed to determining the batch size by a determined forecast. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 12 is non-patent eligible.
Regarding claim 13,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The medium of claim 8, wherein the processing device further performs operations comprising filtering computing instances of the set of computing instances based on a latency threshold.” -- The limitation recites that the processing device further performs operations of filtering computing instances based on a threshold of latency. The limitation amounts to no more than mere instructing the computer/device associated with the computer to perform a generic filtering of data based on a threshold, which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 13 is non-patent eligible.
Regarding claim 14,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The medium of claim 8, wherein the first machine learning model is a regression model.”-- The limitation recites that the first machine learning model stated earlier in above claims is further described as a regression model. The limitation amounts to no more than mere further limiting to a field of use/environment, and thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 14 is non-patent eligible.
Regarding claim 15,
Step 1: The claim is directed to computer-readable medium, which is considered to be an article of manufacture. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The medium of claim 8, wherein the processing device further performs operations comprising training the first machine learning model based on a training dataset including latency obtained from the set of computing instances executing inferencing requests using a set of machine learning models, parameters associated with the set of machine learning models, and FLOPS associated with the set of machine learning models.”-- The limitation recites that the processing device will further performing operations like training the first ML model based on training datasets that also include the latency obtaining from computing instances and other elements that was executed or associated with a ML model(s). The limitation amounts to no more than mere instructions to apply onto a computer, and does not integrate to a practical application, nor provides significantly more than the judicial exception (See MPEP 2106.05(f)).
Thus, claim 15 is non-patent eligible.
Regarding claim 17,
Step 1: The claim is directed to a system, which is considered to be a machine. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The system of claim 16, wherein the information associated with the machine learning model includes at least one of: a number of floating point operations (FLOPs), a number of layers, a number of activations, and a number of parameters.” -- The limitation recites that the information associated with the ML model includes at least a number of FLOPs, layers, activations, and/or parameters. The limitation amounts to no more than mere further limiting to a field of use/environment, and thus does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 17 is non-patent eligible.
Regarding claim 18,
Step 1: The claim is directed to a system, which is considered to be a machine. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The system of claim 16, wherein the latency information indicates an amount of time a first computing instance of the plurality of computing instances takes to process an inferencing request of a first batch size of the plurality of batch sizes.”-- The limitation recites that the latency information will indicate an amount of time a computing instance will take to process an inference request of a first batch out of the group of batch sizes. The limitation amounts to no more than limiting the claim to a field of use/environment, and thus does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 18 is non-patent eligible.
Regarding claim 19,
Step 1: The claim is directed to a system, which is considered to be a machine. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The system of claim 16, wherein the prediction machine learning model is a random forest regression model.” -- The limitation recites that the prediction ML model will further include being described as a random forest regression model. The limitation amounts to no more than merely limiting to a field of use/environment, and thus does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 19 is non-patent eligible.
Regarding claim 20,
Step 1: The claim is directed to a system, which is considered to be a machine. The claim satisfies step 1.
Step 2A Prong 1:
“determining the computing instance and the batch size based on a forecast indicating a number of inferencing requests over an interval of time.” -- The limitation is directed to determining the computing instance and batch size based on the forecast indicating a number of inferencing requests over an interval of time. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with aid of pen and paper, and thus the limitation is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The system of claim 16, wherein causing the latency tool to determine the computing instance and the batch size further comprises” -- The limitation recites causing a latency tool (instructing it) to determine the computing instance and batch size. The limitation amounts to no more than mere instructions to apply onto a computer, and thus it does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 20 is non-patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over NPL reference “INFaaS: Automated model-less inference serving.”, by Romero et. al. (referred herein as Romero) in view of NPL reference “Batch: Machine learning inference serving on serverless platforms with adaptive batching.”, by Ali et. al. (referred herein as Ali) further in view of NPL reference “Improved ML model deployment using Amazon SageMaker Inference Recommender,”, by Kotini et. al. (referred herein as Kotini).
Regarding claim 1, Romero teaches:
A method comprising: obtaining an input to a prediction machine learning model, the input indicating a set of model parameters associated with a machine learning model and a number of floating point operations (FLOPS) associated with performing inferencing operations using the machine learning model; ([Romero, page 400, sec 3.1] “takes a validation dataset (e.g.,valSet) as input...Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID).” and [Romero, page 397, sec 1] “the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets the model accepted by the register_model API, including modelName and modelBinary in serialized format, to be the same as the input indicating a set of model parameters associated with a machine learning model because they are both directed to information identifying and representing a trained machine learning model used for inference. The examiner further interprets computational requirements ranging from 0.48 to 24 GFLOPS to be the same as a number of floating point operations (FLOPS) associated with performing inferencing operations using the machine learning model because they are both directed to floating point computational requirements associated with executing inference using machine learning model variants.)
determining a set of computing instances capable of executing the machine learning model and a set of batch sizes associated with inferencing requests; ([Romero, page 398, sec 2.1] “A model-variant is a version of a model defined by the following aspects: (a) model architecture (e.g., ResNet50, VGG16), (b) programming framework, (e.g., TensorFlow, PyTorch, Caffe2, MXNet), (c) model graph optimizers (e.g., TensorRT, Neuron, TVM, XLA [72]), (d) hyperparameters (e.g., optimizing for batch size of 1, 4, 8, or 16), and (e) hardware platforms (e.g., Haswell or Skylake CPUs, V100 or T4 GPUs, FPGA, and accelerators, such as Inferentia [18] TPU [49] Catapult [32] NPU [7]). Based on the 21 image classification models and the available hardware on AWS EC2 [9] we estimate the total number of possible model-variants would be 4,032.”, wherein the examiner interprets hardware platforms and available hardware on AWS EC2 to be the same as a set of computing instances capable of executing the machine learning model because they are both directed to computing resources available for running machine learning model variants. The examiner further interprets optimizing for batch size of 1, 4, 8, or 16 to be the same as a set of batch sizes associated with inferencing requests because they are both directed to multiple batch-size values used when executing machine learning inference.)
determining a set of weight sum values associated with the set of computing instances based on the latency information; ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j, this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant: Cost(δi j) = Ci j(δi j +λT load i j max(δi j,0)) ”, wherein the examiner interprets the “objective function and cost for an action” to be the same as a set of weight sum values because they are both directed to computed values used to evaluate resource-selection or scaling options. The examiner further interprets the sum of the hardware cost and the loading latency of the variant to be the same as weight sum values based on the latency information because they are both directed to a summed value that includes latency information as part of the calculation.)
Romero does not teach,...causing the prediction machine learning model to output latency information for the set of computing instances and the set of batch sizes, the latency information indicating an interval of time to process an inferencing request of a batch size of the set of batch sizes by a computing instance of the set of computing instances;
Ali teaches:
causing the prediction machine learning model to output latency information for the set of computing instances and the set of batch sizes, ([Ali, page 4, sec 3] “Using as inputs the fitted arrival process, request service times for different system configurations, the monetary budget, and the user SLO, BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies”, wherein the examiner interprets the analytic model implemented within the Performance Optimizer component to be the same as the prediction machine learning model because they are both directed to a model that uses input information to generate predicted latency information. The examiner further interprets “request service times for different system configurations” to be the same as latency information for the set of computing instances and the set of batch sizes because they are both directed to time-based performance information for different inference-serving configurations. Finally, the examiner further interprets “predict the distribution of latencies” to be the same as output latency information because they are both directed to generating latency information for inference-serving configurations.)
the latency information indicating an interval of time to process an inferencing request of a batch size of the set of batch sizes by a computing instance of the set of computing instances; [Ali, page 5, sec 3A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.” and [Ali, page 6, sec 3C] “...where Sk is the service time of a batch of size k.”, wherein the examiner interprets the “time required to process batches of different sizes” to be the same as the latency information indicating an interval of time to process an inferencing request of a batch size because they are both directed to measured time for processing batched inference requests. The examiner further interprets the “serverless platform” and “allocated memory” to be the same as a computing instance of the set of computing instances because they are both directed to computing resources used to execute the inference processing. Finally, the examiner further interprets the service time of a batch of size k to be the same as an interval of time to process an inferencing request of a batch size of the set of batch sizes because they are both directed to the amount of time required to process a batch having a particular batch size.)
Romero and Ali does not teach causing an indication of the set of weight sum values associated with the set of computing instances to be displayed in a user interface.
Kotini teaches causing an indication of the set of weight sum values associated with the set of computing instances to be displayed in a user interface. ([Kotini, sec Inference Recommender job results and metrics] “The results of the default Inference Recommender job contain a list of endpoint configuration recommendations, including instance type, instance count, and environment variables....You can visualize Inference Recommender job results in Amazon SageMaker Studio by choosing Inference Recommender under Deployments in the navigation pane.”, wherein the examiner interprets “endpoint configuration recommendations, including instance type, instance count, and environment variables” to be the same as the set of weight sum values associated with the set of computing instances because they are both directed to recommendation information associated with computing instance configurations. The examiner further interprets “visualize Inference Recommender job results in Amazon SageMaker Studio” to be the same as being displayed in a user interface because they are both directed to presenting recommendation results in a graphical user interface.)
Romero, Ali, Kotini, and the instant application are analogous art because they are all directed to machine learning inference serving and selecting computing resources or configurations based on latency, cost, batch size, and performance considerations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model optimizer technique disclosed by Romero to include the latency prediction techniques disclosed by Ali. One would be motivated to do so to effectively predict latency information across inference-serving configurations and batch sizes for use in selecting an inference-serving configuration, as suggested by Ali ([Ali, page 4, sec 3] “BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies...the Profiler measures the time required to process batches of different sizes”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the model optimizer technique disclosed by Romero to include the user interface display of recommendations disclosed by Kotini. One would be motivated to do so to efficiently present the resulting endpoint configuration recommendations and associated performance information to a user for comparison and deployment selection, as suggested by Kotini ([Kotini, sec Inference Recommender job results and metrics] “The results of the default Inference Recommender job contain a list of endpoint configuration recommendations, including instance type, instance count, and environment variables”; “You can visualize Inference Recommender job results in Amazon SageMaker Studio by choosing Inference Recommender under Deployments in the navigation pane.”).
Regarding claim 2, Romero, Ali and Kotini, teach The method according to claim 1, (see rejection of claim 1).
Romero further teaches wherein the method further comprises selecting a first computing instance of the set of computing instances to execute the machine learning model, where based at least in part on a first weight sum value of the set of weight sum values. ([Romero, page 401, sec 4.1] “When a query arrives, INFaaS’ Dispatcher invokes the getVariant method of model-variant selection policy to choose a variant (Algorithm 1). For this case, the input to getVariant is the query’s requirements, and the output is the variant and worker to serve the query.” and [Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j, this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant: Cost(δi j) = Ci j(δi j +λT load i j max(δi j,0))”, wherein the examiner interprets “choose a variant” and “the output is the variant and worker to serve the query” to be the same as selecting a first computing instance of the set of computing instances to execute the machine learning model because they are both directed to choosing a computing resource, including a worker/hardware-associated variant, to serve an inference query using a machine learning model. The examiner further interprets “the total cost of all the chosen scaling actions” and “the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant” to be the same as a first weight sum value of the set of weight sum values because they are both directed to a computed sum-based value used to evaluate and choose among resource-selection options based at least in part on latency and cost information.)
Regarding claim 3, Romero, Ali and Kotini teach The method according to claim 2, (see rejection of claim 2).
Romero further teaches wherein the first weight sum value is greater than at least one other weight sum value of the set of weight sum values. ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions.”; “Thus, our objective function representing the total cost for all the variants is: ∑i, j Cost(δi j).”; and [Romero, page 405, sec 6.1] “INFaaS achieved higher performance (1.3× higher throughput) and resource utilization (5.6× higher GPU utilization), and lower SLO violations (2× lower) and cost (1.23× lower) compared to the baselines.”, wherein the examiner interprets the objective function representing the total cost for all the variants to be the same as the set of weight sum values because they are both directed to computed comparative values associated with candidate inference-serving resource selections. The examiner further interprets INFaaS achieving higher performance and resource utilization compared to the baselines to be the same as the first weight sum value being greater than at least one other weight sum value of the set of weight sum values because they are both directed to a selected inference-serving configuration having a better comparative selection value than at least one other candidate configuration.)
Regarding claim 4, Romero, Ali and Kotini teach The method according to claim 2, (see rejection of claim 2).
Ali further teaches:
wherein the method further comprises training the prediction machine learning model using a training dataset including a set of metrics obtained by at least causing the set of computing instances to execute a set of machine learning models performing inferencing operations. ([Ali, page 8, sec VI] “We evaluate BATCH with two ML Serving Frameworks...BATCH uses a multivariable regression model that is trained with a few configurations...We evaluate model accuracy and robustness on AWS Lambda using different applications (i.e., MoBiNet, ResNet-v2, and ResNet-18) ” and [Ali, page 7, sec V] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression...the batched requests are processed through the ML model for inference.”, wherein the examiner interprets “multivariable regression model” and “trained with a few configurations” to be the same as training the prediction machine learning model using a training dataset because they are both directed to training a predictive model using configuration-based data. The examiner further interprets “the workload service time,” “different system configurations,” “the batched requests are processed through the ML model for inference,” and “MoBiNet, ResNet-v2, and ResNet-18” to be the same as a training dataset including a set of metrics obtained by causing the set of computing instances to execute a set of machine learning models performing inferencing operations because they are directed to measured service-time metrics obtained from executing batched inference requests using multiple machine learning models under different system configurations.)
Romero, Ali, Kotini, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting inference-serving configurations and using latency or service-time information associated with machine learning model execution.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 2 disclosed by Romero, as modified by Ali and Kotini, to include the “multivariable regression model that is trained with a few configurations” disclosed by Ali. One would be motivated to do so to effectively generate predicted service-time or latency information for additional inference-serving configurations without requiring exhaustive profiling of every possible configuration, as suggested by Ali ([Ali, page 7, sec V] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”).
Regarding claim 5, Romero, Ali and Kotini teach The method according to claim 1, (see rejection of claim 1).
Ali further teaches:
wherein the method further comprises: obtaining a forecast indicating a number of inferencing requests expected over a second interval of time; ([Ali, page 3, sec 2C), “it uses machine learning to predict the arrival intensity, ” and [Ali, page 5, sec 3A] “To determine the workload arrival process, the Profiler observes the inter-arrival times of incoming requests. BATCH uses the KPC-Toolbox [42] to fit the collected arrival trace into a Markovian Arrival Process (MAP) [43] a class of processes that can capture burstiness.” wherein the examiner interprets “predict the arrival intensity” and “inter-arrival times of incoming requests” to be the same as "obtaining a forecast indicating a number of inferencing requests expected over a second interval of time" because they are both the production/use of a model that forecasts the rate (intensity) of incoming inference requests over an upcoming time window; "arrival intensity" is the rate of inference requests per unit time, and predicting/fitting it produces exactly the kind of forward-looking estimate of how many requests are expected over a future interval.)
and determining the batch size of the set of batch sizes to use to process the inferencing requests over the second interval of time based on the latency information. [Ali, page 5, sec 3B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”; and [Ali, page 5, sec 3C] “The Performance Optimizer is the core component of BATCH. It uses the arrival process and service time (both estimated by the profiler) to predict the time required to serve the incoming requests.”, wherein the examiner further interprets “determines the optimal batch size based on the arrival process, service times, SLO, and budget” to be the same as determining the batch size of the set of batch sizes to use to process the inferencing requests because they are both directed to selecting a batch size for processing inference requests. The examiner further interprets “service time” and “predict the time required to serve the incoming requests” to be the same as latency information because they are both directed to time-based performance information for serving inference requests.)
Romero, Ali, Kotini, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting batch-size and computing-resource configurations based on workload demand, latency, and performance considerations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 1 disclosed by Romero, Ali, and Kotini to include the “model and container configurations to improve throughput, reduce latency, and minimize cost” disclosed by Kotini. One would be motivated to do so to efficiently select inference-serving configurations that satisfy expected inference workload demand while improving latency and cost performance, as suggested by Kotini ([Kotini, sec Inference Recommender job results and metrics] “We can use Inference Recommender to benchmark different endpoint configurations in order to optimize our model’s performance.”).
Regarding claim 6, Romero, Ali and Kotini teach The method according to claim 1, (see rejection of claim 1).
Romero further teaches wherein the method further comprises filtering the latency information to remove computing instances of the set of computing instances associated with a latency value over a threshold latency value ([Romero, page 402, sec 4.1] “getVariant first enquires the Metadata Store and retrieves the variant with the lowest combined loading and inference latency that matches the query’s requirements (Line 5). If such a variant is found, INFaaS sends the query to the worker with the lowest utilization on the variant’s target hardware (Lines 6-7). Otherwise, since no registered or generated variant can meet the developer’s requirements, INFaaS suggests a variant that can achieve the closest target accuracy and/or latency (Line 8).”, wherein the examiner interprets retrieving “the variant with the lowest combined loading and inference latency that matches the query’s requirements”, sending “the query to the worker with the lowest utilization” on the variant’s target hardware, and suggesting a variant only when no registered or generated variant can meet the requirements to be the same as filtering the latency information to remove computing instances associated with a latency value over a threshold latency value because they are both directed to using latency requirements to select acceptable computing resources and exclude resources that do not meet the latency requirement.)
Regarding Claim 7, Romero, Ali and Kotini, teach The method according to claim 1, (see rejection of claim 1).
Romero further teaches wherein the set of model parameters associated with the machine learning model and the number of floating point operations (FLOPS) are determined based at least in part on the machine learning model. ([Romero, page 400, sec 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID). Models for different prediction tasks within the same application (e.g., optical character recognition and language translation) can be registered with separate appIDs. Lines 1-2 in Figure 2 show how a developer registers two models, a ResNet50 and a MobileNet, for an application with appID=detectFaceApp. INFaaS generates multiple variants from these already trained models.”, and ([Romero, page 397, sec 1] “These variants vary across many dimensions: the accuracies range from 56.6% to 82.5% (1.46×), the model loading latencies range from 590ms to 11s (18.7×), and the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets the model accepted by the register_model API, including modelName and modelBinary in serialized format, to be the same as the set of model parameters associated with the machine learning model because they are both directed to information identifying and representing the trained machine learning model. The examiner further interprets INFaaS generating multiple variants from already trained models, with the variants having computational requirements ranging from 0.48 to 24 GFLOPS, to be the same as determining the number of floating point operations (FLOPS) based at least in part on the machine learning model because they are both directed to obtaining floating point computational requirements from model-specific variants derived from the machine learning model.)
Claim(s) 8-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Romero in view of Ali.
Regarding claim 8, Romero teaches:
A non-transitory computer-readable medium storing executable instructions embodied thereon, which, when executed by a processing device, cause the processing device to perform operations comprising: ([Romero, page 405, sec 5] “We implemented INFaaS in about 20K lines of C++ code.” and “INFaaS’ API and communication logic between controller and workers are implemented using gRPC in C++ [2].”, wherein the examiner interprets the implemented C++ code and API/communication logic to be the same as executable instructions embodied on a non-transitory computer-readable medium because they are both directed to software instructions that are stored and executed by computing components to perform inference-serving operations.)
the first machine learning model taking as inputs a set of parameters of a second machine learning model and a number of flopping point operations (FLOPs) associated with the second machine learning model; ([Romero, page 400, sec 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID).” and [Romero, page 397, sec 1] “the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets the model accepted by the register_model API, including modelName and modelBinary in serialized format, to be the same as a set of parameters of a second machine learning model because they are both directed to information identifying and representing a trained machine learning model used for inference. The examiner further interprets computational requirements ranging from 0.48 to 24 GFLOPS to be the same as a number of flopping point operations (FLOPs) associated with the second machine learning model because they are both directed to floating point computational requirements associated with executing inference using machine learning model variants.)
determining a computing instance of the set of computing instances and a batch size of the set of batch sizes to execute an inferencing service using the second machine learning model based on the latency information; ([Romero, page 398, sec 2.1] “A model-variant is a version of a model defined by the following aspects: (a) model architecture (e.g., ResNet50, VGG16), (b) programming framework, (e.g., TensorFlow, PyTorch, Caffe2, MXNet), (c) model graph optimizers (e.g., TensorRT, Neuron, TVM, XLA [72]), (d) hyperparameters (e.g., optimizing for batch size of 1, 4, 8, or 16), and (e) hardware platforms (e.g., Haswell or Skylake CPUs, V100 or T4 GPUs, FPGA, and accelerators, such as Inferentia [18] TPU [49] Catapult [32] NPU [7]). Based on the 21 image classification models and the available hardware on AWS EC2 [9] we estimate the total number of possible model-variants would be 4,032.”, wherein the examiner interprets hardware platforms and available hardware on AWS EC2 to be the same as the set of computing instances because they are both directed to computing resources available for running machine learning model variants. The examiner further interprets optimizing for batch size of 1, 4, 8, or 16 to be the same as a set of batch sizes because they are both directed to multiple batch-size values used when executing machine learning inference.)
determining a computing instance of the set of computing instances and a batch size of the set of batch sizes to execute an inferencing service using the second machine learning model based on the latency information; ([Romero, page 401, sec 4.1] “When a query arrives, INFaaS’ Dispatcher invokes the getVariant method of model-variant selection policy to choose a variant (Algorithm 1). For this case, the input to getVariant is the query’s requirements, and the output is the variant and worker to serve the query.” and [Romero, page 401, sec 3.2] “This data consists of (a) the information about available model architectures and their variants (e.g., accuracy and profiled inference latency), and (b) the resource usage and load statistics of variants and worker machines.”, wherein the examiner interprets the output being the variant and worker to serve the query to be the same as determining a computing instance and batch size to execute an inferencing service because they are both directed to selecting a model, hardware, and batch-size configuration to serve an inference query. The examiner further interprets profiled inference latency to be the same as latency information because they are both directed to time-based performance information used for selecting inference-serving configurations.)
causing the computing instance to execute the second machine learning model; ([Romero, page 400, sec 3.1] “The Controller selects a model-variant and dispatches the inference query to a Worker machine. Workers further send inference queries to the appropriate Hardware Executors according to the selected model-variant, and reply with inference results to applications.” and [Romero, page 400, sec 3.2] “Worker machines execute inference queries assigned by the controller. Hardware-specific Executor daemons manage the deployment and execution of model-variants.”, wherein the examiner interprets the Worker machine executing inference queries and Hardware-specific Executor daemons managing deployment and execution of model-variants to be the same as causing the computing instance to execute the second machine learning model because they are both directed to causing a selected computing resource to execute a machine learning model for inference.)
Romero does not teach causing a first machine learning model to predict latency information associated with a set of computing instances executing an inferencing request associated with a set of batch sizes and process inferencing requests including a number of data objects corresponding to the batch size.
Ali teaches:
causing a first machine learning model to predict latency information associated with a set of computing instances executing an inferencing request associated with a set of batch sizes, ([Ali, page 4, sec 3] “Using as inputs the fitted arrival process, request service times for different system configurations, the monetary budget, and the user SLO, BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies”, wherein the examiner interprets the analytic model implemented within the Performance Optimizer component to be the same as the first machine learning model because they are both directed to a model that uses input information to generate predicted latency information. The examiner further interprets “request service times for different system configurations” to be the same as latency information associated with a set of computing instances and a set of batch sizes because they are both directed to time-based performance information for different inference-serving configurations. The examiner further interprets “predict the distribution of latencies” to be the same as predict latency information because they are both directed to generating latency information for inference-serving configurations.)
latency information associated with a set of batch sizes; ([Ali, page 5, sec 3A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.”, wherein the examiner interprets “time required to process batches of different sizes” to be the same as latency information associated with a set of batch sizes because they are both directed to measured time for processing batched inference requests of different batch sizes.)
process inferencing requests including a number of data objects corresponding to the batch size; ([Ali, page 7, sec V] “As soon as the function is invoked, the list of requests are transformed into a batch by the serving function and the batched requests are processed through the ML model for inference.”, wherein the examiner interprets the list of requests transformed into a batch and the batched requests processed through the ML model for inference to be the same as processing inferencing requests including a number of data objects corresponding to the batch size because they are both directed to grouping multiple inference inputs into a batch and processing the batched inputs using the machine learning model.)
Romero, Ali, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting inference-serving configurations and using latency or service-time information associated with machine learning model execution.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the computer-readable medium/software-based inference-serving technique disclosed by Romero to include the latency prediction techniques disclosed by Ali. One would be motivated to do so to effectively predict latency information across inference-serving configurations and batch sizes for use in selecting an inference-serving configuration, as suggested by Ali ([Ali, page 4, sec 3] “BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies”; [Ali, page 5, sec 3A] “the Profiler measures the time required to process batches of different sizes”).
Regarding claim 9, Romero, and Ali teach The medium of claim 8, (see rejection of claim 8).
Romero further teaches wherein the processing device further performs operations comprising: generating a set of weighted sums associated with the set of computing instances and the set of batch sizes based on the latency information; and wherein determining the computing instance is further determined based on the set of weighted sums. ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j, this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant: Cost(δi j) = Ci j(δi j +λT load i j max(δi j,0))”; and [Romero, page 403, sec 4.2.2] “Finally, the model-variant selection policy computes the cost function of our ILP, by using the hardware cost ($/s) and the variant loading latency to decide whether to replicate the running variant, or upgrade to a variant that supports higher throughput.”, wherein the examiner interprets the cost for an action being the sum of the hardware cost and the loading latency of the variant to be the same as generating a set of weighted sums associated with the set of computing instances and the set of batch sizes based on the latency information because they are both directed to computing summed selection values that include latency information for inference-serving resource options. The examiner further interprets using the hardware cost and the variant loading latency to decide whether to replicate the running variant or upgrade to a variant that supports higher throughput to be the same as determining the computing instance based on the set of weighted sums because they are both directed to selecting a computing resource configuration based on computed cost-latency values.).
Regarding claim 10, Romero and Ali teach The medium of claim 9, (see rejection of claim 9).
Romero further teaches wherein the set of weighted sums is determined based on a first weight value associated with latency of the set of computing instances and a second weight value associated with cost of the set of computing instances. ([Romero, page 402, sec 4.2.1] “The objective function that our ILP minimizes is the total cost of all the chosen scaling actions. For a variant vi j, this cost for an action δi j is the sum of the hardware cost (in /second),andtheloadinglatency(inseconds)ofthevariant:Cost(δij)=Cij(δij+λTloadijmax(δij,0))whereCijisthehardwarecost(in/second) for running the variant, T load i j is the loading latency of the variant, and λ (in 1/second) is a tunable parameter for the query load unpredictability. Large values of λ place more weight on minimizing loading latency to meet SLOs when the query load is unpredictable or spiky. Small values of λ place more weight on minimizing the hardware cost when the query load is more stable.”, wherein the examiner interprets “λ” and “place more weight on minimizing loading latency” to be the same as a first weight value associated with latency of the set of computing instances because they are both directed to a weighting factor that changes the contribution of latency in the computed selection value. The examiner further interprets “hardware cost (in $/second) for running the variant” and “place more weight on minimizing the hardware cost” to be the same as a second weight value associated with cost of the set of computing instances because they are both directed to cost information for running a compute resource being included in the computed selection value. The examiner further interprets “this cost for an action δi j is the sum of the hardware cost (in $/second), and the loading latency (in seconds) of the variant” to be the same as the set of weighted sums being determined based on latency and cost because they are both directed to a summed value calculated from latency-related and cost-related factors for selecting an inference-serving resource.).
Regarding claim 11, Romero and Ali teach The medium of claim 8, (see rejection of claim 8).
Ali further teaches wherein the latency information indicates an amount of time computing instances of the set of computing instances take to process an inferencing request of batch sizes of the set of batch sizes using the second machine learning model. ([Ali, page 5, sec 3A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.” and [Ali, page 6, sec 3C] “where Sk is the service time of a batch of size k.”, wherein the examiner interprets “the time required to process batches of different sizes” and “the service time of a batch of size k” to be the same as latency information indicating an amount of time computing instances take to process an inferencing request of batch sizes of the set of batch sizes because they are both directed to time-based performance information for processing batched inference requests. The examiner further interprets the serverless platform and allocated memory to be the same as computing instances of the set of computing instances because they are both directed to computing resources used to execute the inference processing. The examiner further interprets processing batches of different sizes on the serverless platform using an ML serving application to be the same as processing an inferencing request of batch sizes using the second machine learning model because they are both directed to executing batched inference requests using a deployed machine learning model.).
Romero, Ali, and the instant application are analogous art because they are all directed to machine learning inference serving, including determining latency or service-time information for batched inference requests executed using computing resources.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero and Ali to include the “the Profiler measures the time required to process batches of different sizes” disclosed by Ali. One would be motivated to do so to effectively evaluate processing time for different batch-size configurations so that inference-serving configurations can be selected based on latency information, as suggested by Ali ([Ali, page 5, section 3A] “the Profiler measures the time required to process batches of different sizes”).
Regarding claim 12, Romero and Ali teach The medium of claim 8, (see rejection of claim 8).
Ali further teaches wherein the processing device further performs operations comprising: obtaining a forecast predicting a number of inferencing requests over an interval of time; and wherein determining the batch size is further determined based on the forecast. ([Ali, page 5, sec 3A] “To determine the workload arrival process, the Profiler observes the inter-arrival times of incoming requests. BATCH uses the KPC-Toolbox [42] to fit the collected arrival trace into a Markovian Arrival Process (MAP) [43] a class of processes that can capture burstiness.” and [Ali, page 5, sec 3B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”, wherein the examiner interprets determining the workload arrival process using observed inter-arrival times of incoming requests and fitting the collected arrival trace into a Markovian Arrival Process to be the same as obtaining a forecast predicting a number of inferencing requests over an interval of time because they are both directed to modeling the timing/rate of incoming inference requests over time. The examiner further interprets determining the optimal batch size based on the arrival process to be the same as determining the batch size based on the forecast because they are both directed to selecting a batch size using modeled incoming request information.)
Romero, Ali, and the instant application are analogous art because they are all directed to machine learning inference serving, including selecting batch-size and inference-serving configurations based on modeled incoming request information and latency-based performance considerations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero and Ali to include the “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget” disclosed by Ali. One would be motivated to do so to efficiently select a batch size using modeled inference-request arrivals so that the selected batch size satisfies latency and performance goals under bursty or changing workloads, as suggested by Ali ([Ali, page 5, section 3B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”).
Regarding claim 13, Romero and Ali teach The medium of claim 8, (see rejection of claim 8).
Romero further teaches wherein the processing device further performs operations comprising filtering computing instances of the set of computing instances based on a latency threshold. ([Romero, page 401, sec 4.1] “getVariant first enquires the Metadata Store and retrieves the variant with the lowest combined loading and inference latency that matches the query’s requirements (Line 5). If such a variant is found, INFaaS sends the query to the worker with the lowest utilization on the variant’s target hardware (Lines 6-7). Otherwise, since no registered or generated variant can meet the developer’s requirements, INFaaS suggests a variant that can achieve the closest target accuracy and/or latency (Line 8).”, wherein the examiner interprets retrieving the variant with the lowest combined loading and inference latency that matches the query’s requirements and sending the query to the worker with the lowest utilization on the variant’s target hardware to be the same as filtering computing instances of the set of computing instances based on a latency threshold because they are both directed to selecting acceptable inference-serving computing resources based on whether latency satisfies a required latency criterion.)
Regarding claim 14, Romero and Ali teach The medium of claim 8, (see rejection of claim 8).
Ali further teaches wherein the first machine learning model is a regression model. ([Ali, page 8, section 6] “BATCH uses a multivariable regression model that is trained with a few configurations” and [Ali, page 7, section 5] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”, wherein the examiner interprets the multivariable regression model and estimating service times through regression to be the same as the first machine learning model being a regression model because they are both directed to using a regression-based predictive model to estimate latency/service-time information for inference-serving configurations.)
Romero, Ali, and the instant application are analogous art because they are all directed to machine learning inference serving, including predicting latency or service-time information for selecting inference-serving configurations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero and Ali to include the “multivariable regression model that is trained with a few configurations” disclosed by Ali. One would be motivated to do so to effectively estimate service-time or latency information for additional inference-serving configurations without requiring exhaustive profiling of every possible configuration, as suggested by Ali ([Ali, page 7, section 5] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”).
Regarding claim 15, Romero and Ali teach The medium of claim 8, (see rejection of claim 8).
Romero further teaches parameters associated with the set of machine learning models, and FLOPS associated with the set of machine learning models. ([Romero, page 400, section 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID). Models for different prediction tasks within the same application (e.g., optical character recognition and language translation) can be registered with separate appIDs. Lines 1-2 in Figure 2 show how a developer registers two models, a ResNet50 and a MobileNet, for an application with appID=detectFaceApp. INFaaS generates multiple variants from these already trained models.” and [Romero, page 397, section 1] “These variants vary across many dimensions: the accuracies range from 56.6% to 82.5% (1.46×), the model loading latencies range from 590ms to 11s (18.7×), and the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets modelName and modelBinary in serialized format for multiple registered models to be the same as parameters associated with the set of machine learning models because they are both directed to information identifying and representing trained machine learning models. The examiner further interprets computational requirements ranging from 0.48 to 24 GFLOPS to be the same as FLOPS associated with the set of machine learning models because they are both directed to floating point computational requirements associated with executing inference using machine learning model variants.)
Ali further teaches wherein the processing device further performs operations comprising training the first machine learning model based on a training dataset including latency obtained from the set of computing instances executing inferencing requests using a set of machine learning models. ([Ali, page 8, section 6] “BATCH uses a multivariable regression model that is trained with a few configurations” and [Ali, page 7, section 5] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression...the batched requests are processed through the ML model for inference.”, wherein the examiner interprets “multivariable regression model” and “trained with a few configurations” to be the same as training the first machine learning model based on a training dataset because they are both directed to training a predictive model using configuration-based data. The examiner further interprets “the Profiler measures the workload service time under only a few different system configurations” and “the batched requests are processed through the ML model for inference” to be the same as latency obtained from the set of computing instances executing inferencing requests using a set of machine learning models because they are both directed to collecting service-time/latency measurements from computing configurations that execute batched inference requests using machine learning models.)
Romero, Ali, and the instant application are analogous art because they are all directed to training prediction models for machine learning inference serving using latency information, model information, and computational requirements.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero and Ali to include the model-information and computational-requirement features disclosed by Romero. One would be motivated to do so to effectively account for differences among machine learning models when training or applying a latency-prediction model, as suggested by Romero ([Romero, page 400, section 3.1], “INFaaS generates multiple variants from these already trained models.” and [Romero, page 397, section 1], “Their computational requirements range from 0.48 to 24 GFLOPS (50×)”). It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the medium claim 8 disclosed by Romero and Ali to include the “multivariable regression model that is trained with a few configurations” disclosed by Ali. One would be motivated to do so to effectively train a predictive model to estimate latency or service-time information for inference-serving configurations using measured configuration-based performance data, as suggested by Ali ([Ali, page 7, section 5] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”).
Claim(s) 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ali in view of Romero.
Regarding claim 16, Ali teaches:
A system comprising: a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising: obtaining a training dataset including latency information obtained from a plurality of computing instances executing a plurality of inferencing requests of a plurality of batch sizes using a plurality of machine learning models; ([Ali, page 8, section 6] “We evaluate BATCH with two ML Serving Frameworks...BATCH uses a multivariable regression model that is trained with a few configurations...We evaluate model accuracy and robustness on AWS Lambda using different applications (i.e., MoBiNet, ResNet-v2, and ResNet-18)...Fig. 9: CPU and memory usage over three hours of an AWS t2.nano instance hosting BATCH components.”, and [Ali, page 7, section 5] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression ... the batched requests are processed through the ML model for inference.”, wherein the examiner interprets “the Profiler measures the workload service time under only a few different system configurations” to be the same as obtaining a training dataset including latency information obtained from a plurality of computing instances because they are both directed to collecting time-based inference performance data from different computing configurations. The examiner further interprets “the batched requests are processed through the ML model for inference” to be the same as executing a plurality of inferencing requests of a plurality of batch sizes using a plurality of machine learning models because they are both directed to processing batched inference requests using machine learning models. The examiner further interprets MoBiNet, ResNet-v2, and ResNet-18 to be the same as a plurality of machine learning models because they are each machine learning models used to evaluate inference performance.)
training a prediction machine learning model to determine latency for a set of computing instances and a set of batch sizes using the training dataset; ([Ali, page 8, section 6] “BATCH uses a multivariable regression model that is trained with a few configurations” and [Ali, page 7, section 5] “the Profiler measures the workload service time under only a few different system configurations and estimates service times of the remaining ones through regression”, wherein the examiner interprets “multivariable regression model” and “trained with a few configurations” to be the same as training a prediction machine learning model using the training dataset because they are both directed to training a predictive model using configuration-based data. The examiner further interprets estimating service times of the remaining configurations through regression to be the same as determining latency for a set of computing instances and a set of batch sizes because they are both directed to predicting time-based inference performance for different computing and batch-size configurations.)
providing the prediction machine learning model to a latency tool; ([Ali, page 4, section 3] “Using as inputs the fitted arrival process, request service times for different system configurations, the monetary budget, and the user SLO, BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies”, wherein the examiner interprets the analytic model implemented within the Performance Optimizer component to be the same as providing the prediction machine learning model to a latency tool because they are both directed to making a latency-prediction model part of a tool/component that predicts latency information for inference-serving configurations.)
causing the latency tool to determine [computing instance and] a batch size for processing a set of inferencing requests using a machine learning model based at least in part on a result of the prediction machine learning model; ([Ali, page 5, section 3B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.” and [Ali, page 4, section 3] “Using as inputs the fitted arrival process, request service times for different system configurations, the monetary budget, and the user SLO, BATCH uses an analytic model implemented within the Performance Optimizer component to predict the distribution of latencies”, wherein the examiner interprets “The performance optimizer determines the optimal batch size” to be the same as causing the latency tool to determine a batch size because they are both directed to selecting a batch size for processing inference requests. The examiner further interprets “predict the distribution of latencies” to be the same as a result of the prediction machine learning model because they are both directed to outputting predicted latency information used to determine an inference-serving configuration.).
Ali does not teach computing instance and...the prediction machine learning model taking as an input information associated with the machine learning model.
Romero teaches...computing instance and...the prediction machine learning model taking as an input information associated with the machine learning model; ([Romero, page 401, section 4.1] “When a query arrives, INFaaS’ Dispatcher invokes the getVariant method of model-variant selection policy to choose a variant (Algorithm 1). For this case, the input to getVariant is the query’s requirements, and the output is the variant and worker to serve the query.”, wherein the examiner interprets “choose a variant” and “the output is the variant and worker to serve the query” to be the same as causing the latency tool to determine a computing instance because they are both directed to selecting a computing resource to serve an inference query. The examiner further interprets “the input to getVariant is the query’s requirements” to be the same as the prediction machine learning model taking as an input information associated with the machine learning model because they are both directed to using input information associated with the inference query/model requirements to determine an inference-serving configuration.).
Ali, Romero, and the instant application are analogous art because they are all directed to machine learning inference serving and determining inference-serving configurations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the latency prediction techniques disclosed by Ali to include the model selection technique disclosed by Romero. One would be motivated to do so to effectively select a computing resource for serving an inference query based on query or model requirements, as suggested by Romero ([Romero, page 401, section 4.1] “When a query arrives, INFaaS’ Dispatcher invokes the getVariant method of model-variant selection policy to choose a variant (Algorithm 1). For this case, the input to getVariant is the query’s requirements, and the output is the variant and worker to serve the query.”)
Regarding claim 17, Ali and Romero teach The system of claim 16, (see rejection of claim 16).
Romero further teaches wherein the information associated with the machine learning model includes at least one of: a number of floating point operations (FLOPs), a number of layers, a number of activations, and a number of parameters. ([Romero, page 400, section 3.1] “Developers register one or more models using the register_model API. This API accepts a developer-assigned model identifier (modelName), the model (modelBinary) in serialized format (e.g., a TensorFlow SavedModel or model in ONNX format), and a developer-assigned application identifier (appID).” and [Romero, page 397, section 1] “the inference latencies for a single query range from 1.5ms to 5.7s (3,700×). Their computational requirements range from 0.48 to 24 GFLOPS (50×)”, wherein the examiner interprets modelName and modelBinary in serialized format to be the same as information associated with the machine learning model including a number of parameters because they are both directed to information identifying and representing the trained machine learning model. The examiner further interprets computational requirements ranging from 0.48 to 24 GFLOPS to be the same as information associated with the machine learning model including a number of floating point operations because they are both directed to floating point computational requirements associated with executing inference using the machine learning model.).
Ali, Romero, and the instant application are analogous art because they are all directed to machine learning inference serving, including using model-associated information and computational requirements to select or optimize inference-serving configurations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system claim 16 disclosed by Ali and Romero to include the “computational requirements range from 0.48 to 24 GFLOPS” disclosed by Romero. One would be motivated to do so to effectively account for the computational requirements of a machine learning model when determining latency or selecting inference-serving configurations, as suggested by Romero ([Romero, page 397, section 1], “Their computational requirements range from 0.48 to 24 GFLOPS (50×)”).
Regarding claim 18, Ali and Romero teach The system of claim 16, (see rejection of claim 16).
Ali further teaches wherein the latency information indicates an amount of time a first computing instance of the plurality of computing instances takes to process an inferencing request of a first batch size of the plurality of batch sizes. ([Ali, page 5, section 3A] “To profile the inference time on the serverless platform, the Profiler measures the time required to process batches of different sizes for certain amounts of allocated memory.” and [Ali, page 6, section 3C] “where Sk is the service time of a batch of size k.”, wherein the examiner interprets “the time required to process batches of different sizes” and “the service time of a batch of size k” to be the same as latency information indicating an amount of time to process an inferencing request of a first batch size of the plurality of batch sizes because they are both directed to the time required to process batched inference requests having a particular batch size. The examiner further interprets the serverless platform and allocated memory to be the same as a first computing instance of the plurality of computing instances because they are both directed to a computing resource used to execute inference processing.)
Regarding claim 19, Ali and Romero teach The system of claim 16, (see rejection of claim 16).
Ali further teaches wherein the prediction machine learning model is a random forest regression model ([Ali, page 8, section 6-B] “To determine the mean of the service time distribution with as few experiments as possible for the various memory/batch size configurations, BATCH uses a multivariable regression model that is trained with a few configurations (in our experiments, less than 3%). Results show that BATCH can reduce the profiling time by more than 97% compared to exhaustively profiling all system configurations. The accuracy of the regression model is validated against configurations that are not used for training. The mean absolute percentage error is always less than 2%.”, wherein the examiner interprets the multivariable regression model trained with a few configurations to be the same as the prediction machine learning model because they are both directed to a trained predictive model (a random forest regression model is a type of multivariable regression model) used to determine latency or service-time information for inference-serving configurations.).
Regarding claim 20, Ali and Romero teach The system of claim 16, (see rejection of claim 16).
Ali further teaches wherein causing the latency tool to determine the computing instance and the batch size further comprises determining the computing instance and the batch size based on a forecast indicating a number of inferencing requests over an interval of time. ([Ali, page 5, section 3A] “To determine the workload arrival process, the Profiler observes the inter-arrival times of incoming requests. BATCH uses the KPC-Toolbox [42] to fit the collected arrival trace into a Markovian Arrival Process (MAP) [43] a class of processes that can capture burstiness.” and [Ali, page 5, section 3B] “The performance optimizer determines the optimal batch size based on the arrival process, service times, SLO, and budget.”, wherein the examiner interprets determining the workload arrival process using observed inter-arrival times of incoming requests and fitting the collected arrival trace into a Markovian Arrival Process to be the same as a forecast indicating a number of inferencing requests over an interval of time because they are both directed to modeling incoming inference-request activity over time. The examiner further interprets determining the optimal batch size based on the arrival process to be the same as determining the batch size based on the forecast because they are both directed to selecting a batch size using modeled incoming request information.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights