DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending for examination.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim language in the following claims is not clearly understood:
As per claim 1, line 3, 9 and 17, it is unclear whether “inference requests” in line 9 and “ML model inference requests” in line 17 are referring to the same “ML model inference requests” in line 3 (i.e. consistent term should be used with “the” or “said” if they are the same)
As per claims 4 and 15, they have the same deficiency as claim 2 above. Appropriate correction is required.
As per claim 8, line 4-5, it is unclear whether “host usage data” is referring to the same “host usage data” of claim 4. (i.e. consistent term should be used with “the” or “said” if they are the same)
As per claim 17, it has the same deficiency as claim 8 above. Appropriate correction is required.
As per claims 2-3, 5-12, 14-20, they depend from rejected claims and do not resolve the deficiencies thereof and are therefore rejected for at least the same reasons.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kandoi et al. US Pub 2020/0104750 (hereafter Kandoi) in view of Adibowo US Pub 2022/0215008 (hereafter Adibowo) and further in view of Poothia et al. US Pub 2022/0083389 (hereafter Poothia).
As per claim 1, Kandoi teaches the invention substantially as claimed including a computer-implemented method comprising: hosting a machine learning (ML) model endpoint across a first set of hosts (para[0031, 0036, 0042-0043], FIG. 4, ISG group representing first set of hosts hosting machine learning model);
launching a second set of hosts, hosts in the second set of hosts having a software update not included in hosts of the first set of hosts (para[0071-0072], FIG. 10, launch a set of nodes for the new ISG and models, where the new ISG is warmed including loading of artifacts (models and training data from other (not first) ISG) prior to use);
aggregating, from host usage data provided by cache manager applications executing on each host of the first set of hosts, ML model usage data, the ML model usage data including, for a particular ML model (para[0067-0070], metrics are collected, with respect to each host (hosting ML models) including (cache miss rate, host of number of unique models) over the time period);
determining, based on the ML model usage data and the per-ML model priority order of hosts in the second set of hosts, a set of ML models to load to a first host in the second set of hosts (para[0072-0074, 0081-0082], the set of nodes that are healthy are prioritized (based on the received health information), and are provided to new ISG to load the ML model);
causing the first host to load the set of ML models (para[0072-0074], load at ML models onto the new ISG);
and updating the router to direct ML model inference requests amongst the second set of hosts (para[0056, 0083-0084], the health status of the hosts are updated for the load balancer, and route the requests to the register target (hosts) that are healthy).
Kandoi does not explicitly teach wherein a router directs model ML inference requests amongst the first set of hosts in a prioritized order based on a hash function; a number of inference requests to the particular ML model and a time that the particular ML model was last requested; calculating, using the hash function, a per-ML model priority order of hosts in the second set of hosts;
However, Adibowo teaches a router directs model ML inference requests amongst the first set of hosts in a prioritized order based on a hash function; calculating, using the hash function, a per-ML model priority order of hosts in the second set of hosts (para[0007, 0045-0051], FIG. 3 and 4, selecting a model server by sorting the list of model server (rank order highest to lowest) based on the hash value to send the inference request).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Adibowo’s teaching to Kandoi’s invention in order to provide a machine learning inference platform for large scale, multi-model ML inference services which enables adjusting a number of servers in use for ML model inference dynamically based on the usage to balance user experience and service cost (para[0005, 0030]).
Kandoi and Adibowo do not explicitly teach a number of inference requests to the particular ML model and a time that the particular ML model was last requested.
However, Poothia teaches a number of inference requests to the particular ML model and a time that the particular ML model was last requested (para[0025-0027, 0063], candidate hardware resources each has a score, and the execution priorities associated with hardware resources/nodes are calculated based on inference request count on a node in last x hours, thus the inference requests count with time for the ML model is recorded for the usage data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Poothia’s teaching to Kandoi and Adibowo’s invention in order to provide a method for efficiently scheduling a machine learning model to hardware resources based on an AI inference service by comparing the computational workload of the machine learning model with the computational abilities and functions of the hardware resources, where the hardware resource utilization is efficient across the cluster and provides significant improvements to latencies for the machine learning workloads (para[0004, 0043, 0052]).
As per claim 2, Kandoi, Adibowo and Poothia teach the computer-implemented method of claim 1, and Kandoi teaches wherein hosts in the first set of hosts and in the second set of hosts are virtual machines, and wherein the router is a routing application (para[0028, 0045, 0056], the resources are VMs, and the router is a virtual router (application)).
As per claim 3, Adibowo teaches wherein the hash function generates an output within a hash space based on an ML model identifier and a host identifier, and wherein the priority order of hosts for a first ML model is based on an order of hash function outputs within the hash space for an ML model identifier of the first ML model and each host identifier of hosts in the second set of hosts (para[0007, 0045-0051], FIG. 3 and 4, selecting model server is based on hashing a combination of model server identifier and the ML model identifier and ordering the model servers based on respective hash values).
As per claim 4, Kandoi teaches the invention substantially as claimed including a computer-implemented method comprising: aggregating, from host usage data provided from each host of a first set of hosts, machine learning (ML) model usage data (para[0067-0070], metrics are collected, with respect to each host (hosting ML models) including (cache miss rate, host of number of unique models) over the time period);
determining, based on the ML model usage data and the priority order, a set of ML models to load to a particular host in the second set of hosts (para[0072-0074, 0081-0082], the set of nodes that are healthy are prioritized (based on the received health information), and are provided to new ISG to load the ML model);
causing the particular host to load the set of ML models (para[0072-0074], load at ML models onto the new ISG);
and updating a router to direct ML model inference requests amongst the second set of hosts (para[0083-0084], the health status of the hosts are updated for the load balancer, and route the requests to the register target (hosts) that are healthy).
Kandoi does not explicitly teach the ML model usage data including, for a particular ML model, a number of inference requests to the particular ML model; calculating a priority order of hosts in a second set of hosts to service an inference request for the particular ML model.
However, Adibowo teaches calculating a priority order of hosts in a second set of hosts to service an inference request for the particular ML model (para[0007, 0045-0051], FIG. 3 and 4, selecting a model server by sorting the list of model server (rank order highest to lowest) based on the hash value to send the inference request).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Adibowo’s teaching to Kandoi’s invention in order to provide a machine learning inference platform for large scale, multi-model ML inference services which enables adjusting a number of servers in use for ML model inference dynamically based on the usage to balance user experience and service cost (para[0005, 0030]).
Kandoi and Adibowo do not explicitly teach the ML model usage data including, for a particular ML model, a number of inference requests to the particular ML model.
However, Poothia teaches the ML model usage data including, for a particular ML model, a number of inference requests to the particular ML model (para[0025-0027, 0063], candidate hardware resources each has a score, and the execution priorities associated with hardware resources/nodes are calculated based on inference request count on a node in last x hours, thus the inference requests count with time for the ML model is recorded for the usage data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Poothia’s teaching to Kandoi and Adibowo’s invention in order to provide a method for efficiently scheduling a machine learning model to hardware resources based on an AI inference service by comparing the computational workload of the machine learning model with the computational abilities and functions of the hardware resources, where the hardware resource utilization is efficient across the cluster and provides significant improvements to latencies for the machine learning workloads (para[0004, 0043, 0052]).
As per claim 5, it is a computer-implemented method claim of claim 2 above, thus it is rejected for the same rationale.
As per claim 6, it is a computer-implemented method claim of claim 3 above, thus it is rejected for the same rationale.
As per claim 7, Kandoi, Adibowo and Poothia teach the computer-implemented method of claim 6, and Adibowo teaches wherein the router uses the hash function to identify a host amongst the second set of hosts to which to route a received ML model inference request (para[0007, 0045-0051], FIG. 3 and 4, selecting model server is based on hashing a combination of model server identifier and the ML model identifier and ordering the model servers based on respective hash values).
As per claim 8, Kandoi teaches wherein each host in the first set of hosts executes a cache manager application to load ML models (1) from a network storage location to a host storage location and (2) from the host storage location to a memory of the host associated with an environment to execute a ML model inference (para[0031-0032, 0049-0050, 0057], ML model acquired from ML model storage and load the models to model cache).
and further comprising requesting host usage data from the cache manager applications of the first set of hosts (para[0067-0070], metrics are collected, with respect to each host (hosting ML models) including (cache miss rate, host of number of unique models) over the time period).
As per claim 9, Kandoi teaches wherein the environment to execute the ML model inference is a container (para[0028-0029], compute resources are containers which executes the ML model inference).
As per claim 10, Kandoi teaches wherein a cache manager application of the particular host loads the set of ML models from the network storage location to a memory of the particular host associated with an environment to execute the set of ML models (para[0031-0032, 0049-0050, 0057], ML model acquired from ML model storage and load the models to model cache).
As per claim 11, Kandoi teaches wherein the first set of hosts are part of an ML model hosting endpoint before the router update and the second set of hosts form the ML model hosting endpoint after the router update (para[0031, 0036, 0042-0043, 0071-0072], FIG. 4, ISG group 1 and 2 representing first and second set of hosts hosting machine learning model, where the router is updated to route to the second set of hosts after launching);
and further comprising receiving a request to apply a software update to at least one of the ML model hosting endpoint or a host in the first set of hosts (para[0071-0072], FIG. 10, launch a set of nodes for the new ISG and models, where the new ISG is warmed including loading of artifacts (models and training data (software update) prior to use).
As per claim 12, Kandoi teaches wherein after updating the router, the router no longer directs ML model inference requests to the first set of hosts (para[0056, 0083-0084], the router is updated regarding the inference service groups (updating the whitelist configuration), and route the requests to the register target (hosts) that are healthy).
As per claim 13, it is a system claim of claim 4 above, thus it is rejected for the same rationale.
As per claim 14, it is a system claim of claim 5 above, thus it is rejected for the same rationale.
As per claim 15, it is a system claim of claim 6 above, thus it is rejected for the same rationale.
As per claim 16, it is a system claim of claim 7 above, thus it is rejected for the same rationale.
As per claim 17, it is a system claim of claim 8 above, thus it is rejected for the same rationale.
As per claim 18, it is a system claim of claim 10 above, thus it is rejected for the same rationale.
As per claim 19, it is a system claim of claim 11 above, thus it is rejected for the same rationale.
As per claim 20, it is a system claim of claim 12 above, thus it is rejected for the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Goli US Pub 2020/0356415 teaches edge system of an Internet of Things system including a memory configured to store a machine learning (ML) model application having a ML model a machine, and a processor configured to cause a ML inference service to receive a request for an inference from a ML model application having a ML model, and load the ML model application from the memory into an inference engine in response to the request.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAMMY EUNHYE LEE whose telephone number is (571)270-7773. The examiner can normally be reached Mon, Tues, Thur 9PM-4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aimee Li can be reached at (571)272-4169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAMMY E LEE/Primary Examiner, Art Unit 2195
Prosecution Insights