DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on February 2nd, 2026 has been entered.
Response to Arguments
Applicant argues that Chen fails to disclose “loading…an ML model application…into the inference engine at the second system” and further fails to disclose “select[ing], by the inference engine at the second system, a runtime environment from the ML model application…based on the hardware configuration of the second system (Remarks, pg. 9-10).”
The Examiner agrees that Chen does not expressly describe the inference engine at the second system selecting a runtime environment from the deployed ML model application based on the hardware configuration of the second system. Accordingly, Chen is not relied upon alone for this limitation.
However, Applicant’s characterization of Chen is otherwise inconsistent with the disclosure of the reference. Chen expressly teaches deployment of machine learning models to heterogeneous edge devices based upon their respective hardware capabilities. Chen further discloses generation and optimization of machine learning models for target hardware, deployment of containerized machine learning applications, loading model instances into an inference engine at the edge device, and execution of inference by that inference engine. See, e.g., Chen, col. 3, ll. 43-60; col. 5, ll. 47-63; col. 6, ll. 1-67; col. 10, ll. 39-53; Figs. 1-3. Thus, Chen is properly relied upon for the claimed generation, deployment, loading, and execution of the machine learning model application at the second system.
Further, Chen generally describes the use of an IoT system, even if the term “IoT” or “Internet of Things” is not explicitly used (Remarks, pg. 12-13). Specifically, Chen recites: “The one or more edge devices 122A-122N may then operate as intended, e.g., by an application 127 capturing/creating input data via one or more sensors 128 (e.g., optical sensors, audio sensors, temperature sensors, humidity sensors, air pressure sensors, gas sensors, moisture sensors, water flow sensors, weight sensors, motion sensors, global positioning system (GPS) sensors, rotation/acceleration sensors, radio sensors, biological sensors (e.g., pulse sensors), fingerprint sensors, and the like… (Chen, col. 7:56-64).” This corresponds to the definition of IoT as “Internet of things (IoT) describes physical objects that are embedded with sensors, processing ability, software, and other technologies that connect and exchange data with other devices and systems over the Internet or other communication networks.” See attached NPL: Internet of Things, Wikipedia.
As set forth in the rejection, Koblents teaches selecting an execution runtime or execution environment at the target system based upon the hardware configuration of that system. It would have been obvious to one of ordinary skill in the art to incorporate the runtime-selection technique of Koblents into Chen’s edge inference architecture in order to improve execution portability and performance across heterogeneous hardware platforms while preserving Chen’s objective of supporting deployment to edge devices having differing hardware resources. The proposed modification merely determines, at execution time, which available runtime environment should execute the deployed machine learning application and represents the predictable use of known runtime-selection techniques to improve Chen’s expressly disclosed heterogeneous deployment framework.
Accordingly, the combination of Chen and Koblents teaches or renders obvious the disputed limitations, and the rejection is maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 9-16, 25-32, and 36-49 are rejected under 35 U.S.C. 103 as being unpatentable under US Patent 11,301,762 (Chen et al; Chen) in view of US Pre-Grant Patent 2019/0196853 (Koblents et al; Koblents).
Regarding claim 9 and analogous claims 25 and 36:
Chen teaches:
1. At least one non-transitory computer-readable storage medium including instructions that, when executed by a processor of a second system, cause the processor of the second system to: receive, at a machine learning (ML) inference service, a request for an inference, a prediction, or a combination thereof, using an ML model;
(Chen, col. 3:36-39)
“…developers can simply provide a trained model, send a simple request to deploy it to one or more electronic devices (which may or may not be homogeneous) [i.e. At least one non-transitory computer-readable storage medium including instructions that, when executed by a processor of a second system, when executed by a processor of a second system, cause the processor of the second system to:] and use a pre-defined API in their applications to perform inference [i.e. receive, at a machine learning (ML) inference service, a request for an inference, using an ML model;).”
2. load, into the inference engine at the second system, an ML model application that includes the ML model in response to the request, wherein the ML model application comprises at least one container, and wherein the ML model application is generated at a first computing system from a core ML model based on a processing hardware capability of the second system hosting the inference engine on which the ML model application is to be deployed;
(Chen, col. 3: 43-57, Fig. 1)
“FIG. 1 is a diagram illustrating an exemplary environment including a unified inference framework (“UIF”) server module 112 according to some embodiments. The UIF server module 112, in some embodiments, is a portion of software allowing users 118 to deploy and manage high performance machine learning models 130 running on connected devices 122 in production [i.e. load, into the inference engine at the second system, an ML model application that includes the ML model in response to the request, wherein the ML model application comprises at least one container]. Users 118 (e.g., individuals, organizations, even OEMs) can import to—or train machine learning models in—a provider network 100 (“the cloud”), and reliably deploy these models to large numbers of devices 122 at the edge. Embodiments can—via a model optimizer 114 of a UIF server module 112 and/or UIF client module 124—automatically tune ML models for optimal performance across multiple underlying hardware platforms [i.e. and wherein the ML model application being generated from a core ML model based on a processing hardware capability of the second system hosting the inference engine on which the ML model application is to be deployed;],”
3. select, by the inference engine at the second system, [a runtime environment from the ML model application to execute the ML model, the selection based on the hardware configuration of the system;]
(Chen, col. 7: 64; col. 8: 1-5).
“The inference engine 132 itself, or another application 127 executed by the one or more edge devices 122A-122N, at optional circle (7) may perform actions based on the inference values (or when the inference values satisfy some condition, e.g., when they exceed a threshold value) [i.e. select, by the inference engine at the second system].”
4. cause the ML model to be executed by the ML model application loaded in the inference engine using the selected runtime environment to provide an inference result, a prediction result, or a combination thereof;
(Chen, col. 7:64-67; col. 8:1-5)
“This input data, at circle (5), is provided to the inference engine 132 which at circle (6) can perform inference using the optimized model 130 and optionally logic of an inference library 134 [i.e. cause the ML model to be executed by the ML model application loaded in the inference engine]. The inference engine 132 itself, or another application 127 executed by the one or more edge devices 122A-122N, at optional circle (7) may perform actions based on the inference values (or when the inference values satisfy some condition, e.g., when they exceed a threshold value) [i.e. using the selected runtime environment to provide an inference result, a prediction result, or a combination thereof;].”
5. and provide the inference result, the prediction result, or a combination thereof, at an output of the inference engine.
(Chen, col. 8:10-17)
As another example, the input data or inferences could be sent to a machine learning service, sent to a data monitoring/logging service, stored in a database, sent to a serverless code execution service to be processed, etc., allowing users to take “local” inference results generated by edge devices and integrate these results into an overall application 127 in nearly any manner desired by the users [i.e. and provide the inference result, the prediction result, or a combination thereof, at an output of the inference engine].
Chen does not explicitly teach:
1. [select, by the inference engine at the second system,] a runtime environment from the ML model application to execute the ML model, the selection based on the hardware configuration of the system;
Koblents teaches:
1. [select, by the inference engine at the second system,] a runtime environment from the ML model application to execute the ML model, the selection based on the hardware configuration of the system;
(Koblents, ¶0019, Fig. 4)
“Embodiments of the present invention include one or more programs of a runtime system that profile and store performance data of programs, when executed separately by both a CPU and a GPU, utilizing low overhead, and predict whether to use the CPU or the GPU to execute the program more quickly [i.e. a runtime environment from the ML model application to execute the ML model, the selection based on the hardware configuration of the system;].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. Chen teaches an inference engine at the second system configured to execute the deployed ML model. Koblents teaches selecting an execution environment based on the available hardware resources prior to execution. It would have been obvious to modify Chen’s inference engine to perform Koblents’ runtime-selection technique before executing the deployed ML model because such a modification predictably improves execution across heterogeneous hardware platforms.
Regarding claim 10 and analogous claims 26 and 39:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the instructions further cause the system to select the runtime environment associated with a first execution hardware component in response to a second execution hardware component being unavailable.
(Chen, col. 12:33-41)
“The model training system 620 can automatically scale up and down based on the volume of training requests received from user devices 602 [i.e. wherein the instructions further cause the system to select the runtime environment associated with a first execution hardware component] via frontend 629 [i.e. a second execution hardware component] [i.e. thereby relieving the user from the burden of having to worry about over-utilization (e.g., acquiring too little computing resources and suffering performance issues) or under-utilization (e.g., acquiring more computing resources than necessary to train the machine learning models, and thus overpaying) [i.e. being unavailable].”
Examiner notes that if the system is over-utilized, the second execution hardware components would not be available.
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 11 and analogous claims 27 and 40:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the instructions further cause the second system to load the ML model application into an inference engine in response to a determination that the ML model application is unavailable in other inference engines.
(Chen, col. 8: 58-67; col. 9:1-8)
“As shown at circle (A1), in some embodiments a ML model 108 in a format pertaining to a particular framework can be provided to a UIF server module 112 that can perform translation of the model into another common format, which may be sent as model 129 at circle (B) to be optimized by each on-device model optimizer 114B and then used by the inference engine 132 [i.e. wherein the instructions further cause the second system to load the ML model application into an inference engine]. Alternatively, the UIF server module 112 can perform translation of the model into another common format and also (partially) optimize the model, resulting in a translated and partially-optimized model 129 that may be sent at circle (B) to be optimized by each on-device model optimizer 114B and then used by the inference engine 132 as model 130 [i.e. in response to a determination that the ML model application is unavailable in other inference engines].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 12 and analogous claims 28 and 41:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the instructions further cause the second system to map the ML model application to the inference engine.
(Chen, col. 8:40-48)
“As shown, a number of framework-specific models 204 generated using a number of machine learning frameworks 202 (e.g., MXNet, TensorFlow, Caffe, etc.) can be provided to one or more model optimizers 114A/114B to be translated and optimized, resulting in an optimized model being provided to an inference engine 132 that can run the optimized model with input data 206 generated using hardware resources 126 (such as one or more sensors 128) to generate inferences 208 [i.e. wherein the instructions further cause the second system to map the ML model application to the inference engine].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 13 and analogous claims 29 and 42:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the instructions further cause the second system to direct a second request for an inference, a prediction, or a combination thereof, from the ML model application to the inference engine in response to a determination that the ML model application is mapped to the inference engine.
(Chen, col. 8:65-67, col. 9:1-3)
“Alternatively, the UIF server module 112 can perform translation of the model into another common format and also (partially) optimize the model, resulting in a translated and partially-optimized model 129 that may be sent at circle (B) to be optimized by each on-device model optimizer 114B and then used by the inference engine 132 as model 130 [i.e. wherein the instructions further cause the system to direct a second request for an inference... from the ML model application to the inference engine in response to a determination that the ML model application is mapped to the inference engine.)”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 14 and analogous claims 30 and 43:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the instructions further cause the second system to select a different runtime environment for execution of the second request.
(Chen, col. 9:19-24)
“Alternatively, the UIF server module 112 can perform translation of the model into another common format and also partially or completely optimize the model [i.e. wherein the instructions further cause the system to select a different runtime environment], resulting in an optimized model that may be sent at circle (C) to be used by the inference engine 132 of each edge device without on-device optimizations [i.e. for execution of the second request].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 16 and analogous claims 32 and 45:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the instructions further cause the second system to receive a version of the ML model with the request.
(Chen, col. 9:19-24)
“Alternatively, the UIF server module 112 can perform translation of the model into another common format and also (partially) optimize the model, resulting in a translated and partially-optimized model 129 that may be sent at circle (B) to be optimized by each on-device model optimizer 114B and then used by the inference engine 132 as model 130 [i.e. wherein the instructions further cause the system to receive a version of the ML model with the request].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 31 and analogous claim 44:
Chen and Koblents teach the system of claim 25.
Chen further teaches:
1. wherein the processor is further configured to receive an identifier associated with the ML model with the request.
(Chen, col. 11:23-37)
“The user devices 602 can interact with the model training system 620 via frontend 629 of the model training system 620. For example, a user device 602 can provide a training request to the frontend 629 that includes a container image (or multiple container images, or an identifier of one or multiple locations where container images are stored), an indicator of input data (e.g., an address or location of input data), one or more hyperparameter values (e.g., values indicating how the algorithm will operate, how many algorithms to run in parallel, how many clusters into which to separate data, etc.), and/or information describing the computing machine on which to train a machine learning model (e.g., a graphical processing unit (GPU) instance type, a central processing unit (CPU) instance type, an amount of memory to allocate, a type of virtual machine instance to use for training, etc.) [i.e. wherein the processor is further configured to receive an identifier associated with the ML model with the request].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 37:
Chen and Koblents teach the method of claim 36.
Chen teaches:
1. wherein the ML model application is generated by an ML inference generation tool, and based at least on the ML model being configured based on the hardware configuration of the second system.
(Chen, col. 3:53-60)
“Embodiments can—via a model optimizer 114 of a UIF server module 112 and/or UIF client module 124—automatically tune ML models for optimal performance across multiple underlying hardware platforms [i.e. wherein the ML model application is generated by an ML inference generation tool], resulting in improved prediction/inference speeds that allow sophisticated computer vision, audio, and anomaly detection models to run efficiently, even on low-power devices [i.e. and based at least on the ML model being configured based on the hardware configuration of the second system].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 38:
Chen and Koblents teach the method of claim 36.
Chen teaches:
1. wherein the ML model application is deployed to the second system by a centralized manager of the first computing system
(Chen, col. 4: 49-53)
“According to some embodiments, a user 118 may utilize a client device (e.g., client device 120) such as a Personal Computer (PC), laptop or other mobile device such as a smartphone, tablet, etc., to manage the deployment of a ML model to one or more edge devices 122A-122N [i.e. wherein the ML model application is deployed to the second system by a centralized manager of the first computing system].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Regarding claim 47 and analogous claims 48 and 49:
Chen and Koblents teach the non-transitory computer-readable storage medium of claim 9.
Chen further teaches:
1. wherein the system is an edge system of an Internet of Things (IoT) system
(Chen, col. 4: 54-56)
“Ones of the edge devices 122A-122N may be preconfigured (e.g., at or after manufacture time, such as by an OEM or other provider) with a UIF client module 124 [i.e. wherein the system is an edge system of an Internet of Things (IoT) system].”
2. and wherein the first computing system is a centralized Internet of Things (IoT) computing system.
(Chen, col. 3:43-53)
“FIG. 1 is a diagram illustrating an exemplary environment including a unified inference framework (“UIF”) server module 112 according to some embodiments. The UIF server module 112, in some embodiments, is a portion of software allowing users 118 to deploy and manage high performance machine learning models 130 running on connected devices 122 in production. Users 118 (e.g., individuals, organizations, even OEMs) can import to—or train machine learning models in—a provider network 100 (“the cloud”), and reliably deploy these models to large numbers of devices 122 at the edge [i.e. and wherein the first computing system is a centralized Internet of Things (IoT) computing system.].”
One of ordinary skill in the art, at the time the invention is filed, would have been motivated to modify Chen with Koblents. The motivation is the same as claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL JUSTIN BREENE whose telephone number is (571)272-6320. Examiner
interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-
based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO
Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on 303-297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786 9199 (IN USA OR CANADA) or 571-272-1000.
/P.J.B./ Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights