Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
Examiner cites particular columns and line numbers in the references as applied to the claims below for convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references cited in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Claim Objections
Claims 1, 4-5, 7, 9-11, 14-16, and 18-20 are objected to because of the following informalities:
In claim 1, line 9, it is unclear what is meant by “an amount of processing devices of the device plugin” as a device plugin does not typically comprise processing devices. The Examiner has interpreted the recited “amount of processing devices of the device plugin” to be referring to an “amount of processing devices of the portion of the plurality of processing devices reserved by [[of]] the device plugin” or just an “amount of processing devices of the portion of the plurality of processing devices [[device plugin]]”.
In claim 4, line 7, it is unclear what is meant by “a second amount of processing devices of the device plugin” as a device plugin does not typically comprise processing devices. The Examiner has interpreted the recited “second amount of processing devices of the device plugin” to be referring to a “second amount of processing devices of the portion of the plurality of processing devices reserved by [[of]] the device plugin” or just a “second amount of processing devices of the portion of the plurality of processing devices [[device plugin]]”.
In claim 5, line 3, “the first and second amount” should recite “the first amount and the second amount”.
In claim 7, line 12, a comma should be added after interface (“by the container runtime interface,”).
In claim 9, line 2, “the request” is unclear whether it is referring to the request for instantiation of a container (see claim 1, line 8), the processor request for an amount of processing devices of the device plugin (see claim 1, lines 8-9), or the request by the orchestrator which requests allocation of the amount by the device plugin (see claim 1, line 10).
For the same reasons, “the request” as later recited in claim 9, line 4, and claim 10, lines 2-3, is unclear and should be clarified as well.
In claim 11, lines 6-7, it is unclear what is meant by “an amount of processing devices of the device plugin” as a device plugin does not typically comprise processing devices. The Examiner has interpreted the recited “amount of processing devices of the device plugin” to be referring to an “amount of processing devices of the portion of the plurality of processing devices reserved by [[of]] the device plugin” or just an “amount of processing devices of the portion of the plurality of processing devices [[device plugin]]”.
In claim 14, lines 6-7, it is unclear what is meant by “a second amount of processing devices of the device plugin” as a device plugin does not typically comprise processing devices. The Examiner has interpreted the recited “second amount of processing devices of the device plugin” to be referring to a “second amount of processing devices of the portion of the plurality of processing devices reserved by [[of]] the device plugin” or just a “second amount of processing devices of the portion of the plurality of processing devices [[device plugin]]”.
In claim 15, lines 1-2, “the first and second amount” should recite “the first amount and the second amount”.
In claim 16, line 11, a comma should be added after interface (“by the container runtime interface,”).
In claim 18, line 2, “the request” is unclear whether it is referring to the request for instantiation of a container (see claim 11, lines 5-6), the processor request for an amount of processing devices of the device plugin (see claim 11, line 6), or the request by the orchestrator which requests allocation of the amount by the device plugin (see claim 11, line 8).
For the same reasons, “the request” as later recited in claim 18, line 3, and claim 19, line 1, is unclear and should be clarified as well.
In claim 20, line 6, it is unclear what is meant by “an amount of processing devices of the device plugin” as a device plugin does not typically comprise processing devices. The Examiner has interpreted the recited “amount of processing devices of the device plugin” to be referring to an “amount of processing devices of the portion of the plurality of processing devices reserved by [[of]] the device plugin” or just an “amount of processing devices of the portion of the plurality of processing devices [[device plugin]]”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 4-5, 7-12 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over MEHROTRA et al. (U.S. Pub. No. 2024/0344170), hereinafter MEHROTRA, in view of ZHAO et al. (U.S. Pub. No. (U.S. Pub. No. 2026/0010511), hereinafter ZHAO.
Regarding claim 1, MEHROTRA teaches A system comprising:
a computing device including a plurality of processing devices and one or more memory devices operably coupled to the plurality of processing devices, the one or more memory devices storing executable code that, when executed by the plurality of processing devices, causes the plurality of processing devices ([0222]-[0224] – “a server that implements a portion or all of one or more of the technologies described herein, including the techniques to implement the capturing of network traffic may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media. FIG. 9 illustrates such a general-purpose computing device 900. In the illustrated embodiment, computing device 900 includes one or more processors 910a, 910b, and/or 910n (which may be referred herein singularly as "a processor 910" or in the plural as "the processors 910") coupled to a system memory 920 […] a multiprocessor system including several processors 910 (e.g., two, four, eight, or another suitable number). Processors 910 may be any suitable processors capable of executing instructions. […] System memory 920 may be configured to store instructions and data accessible by processor(s) 910 […] program instructions and data implementing one or more desired functions, such as those methods, techniques and data described above, are shown stored within system memory 920 as code 929 and data 926.”) to:
reserve a portion of the plurality of processing devices by a device plugin, the portion including at least two processing devices of the plurality of processing devices ([0006]-[0008] – “Plugins can be used to identify and manage available resources, improve the performance of workloads, and provide a framework for managing and scheduling these resources as well as utilizing the resources by the workload. In an example, isolated CPU cores are advertised as separate resources. A number of CPU cores to be isolated are identified, such that when the system boots up, these isolated cores are not available for general use. The orchestration system is notified so that when resources are advertised, the isolated cores are not generally advertised in the same way as other resources. Rather, the isolated cores are advertised as available for eligible workloads. Instead of having the orchestration system advertise isolated cores, device plugins are used to advertise these isolated cores. In an embodiment, the device plug-in is configured to manage and allow eligible workloads to use the isolated cores. The device plug-in excludes the isolated cores from OS scheduling as well as scheduling by the orchestration system. In addition to isolating a number of CPU cores, the disclosed techniques allow for management of the isolated resources. […] In an embodiment, scheduler plugins of an orchestration system can be used to manage resource sets. The scheduler plugins can be used to manage resource status and availability.”; [0158] – “For other resources such as CPUs, resources can be removed from the standard advertisement pool and assigned to other resources. For example, a node may have 32 CPUs, where 24 are isolated CPU cores. Those 24 isolated cores can be reserved so that the node is not allowed to advertise them as standard CPUs. Instead, the device plugin is used to advertise the 24 isolated cores as separate resources. In this way, standard resources are partitioned into subclasses so that container workloads which require the resources can request them.”);
receive, by an orchestrator, a request for instantiation of a container including a processor request for an amount of processing devices of the device plugin ([0009] – “The disclosure also provides a way to launch a given workload when an isolated resource is being used and to ensure that the isolation is maintained. In an embodiment, when a node launches the workload, the runtime call is intercepted”; [0045] – “the number of each resource that is needed in the container workload can be requested as resource requests. The requested resources can be scheduled onto a node where there are a sufficient number of resources”; [0096] – “The above request can be interpreted as "requests" being an array of resource requests for the workload, with "requests.[i].type" representing the type of resource being requested, and "requests[i].quantity" being the number of these resources being requested. "requests.[i].features" is an array of potential features which qualify as matching the request.”; [0156] – “In response to receiving a request 132B from one of the computing nodes 102 to use one of the resources, the orchestrator reserves the requested resource for use by the computing node 102 that sent the request 132B.”; [0187]-[0188] – “the system is extended by utilizing a custom runtime which intercepts the default runtime when a container is instantiated. At this point, the container configuration is examined and modified so that device and resource visibility is limited. In addition, any resources being used by the container can be configured. Additionally, the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized.”; [0197] – “The disclosed embodiments enable partitioning of resources/devices into different classes for resource scheduling purposes using a feature or characteristic. CPUs in particular can be partitioned into isolcpus and non-isolcpus”; [0214] – “each of the resource classes are associated with a set of characteristics. In an embodiment, the characteristics include whether an associated resource is isolated or non-isolated”; [0215] – “in response to receiving, from a workload running in one of the containerized functions, a request to use one of the resource classes, selecting one of the computing resources that is available for use by the requesting workload based on a set of characteristics associated with the one of the resource classes requested by the requesting workload.”);
[…], allocation of the amount by the device plugin ([0007] – “the device plug-in is configured to manage and allow eligible workloads to use the isolated cores. The device plug-in excludes the isolated cores from OS scheduling as well as scheduling by the orchestration system.”; [0157] – “The device plugins 153 advertise other resources which can be used to extend the resources to non-native Kubernetes resources. A custom container runtime interface (CRI) 165, as further discussed below, is used to provide a configuration to ensure that the correct resources are used by containers during container launch 170. Device plugins 153 are used to advertise resource sets and to manage and allow eligible workloads to use the resource sets.”; [0158]-[0159] – “a node may have 32 CPUs, where 24 are isolated CPU cores. Those 24 isolated cores can be reserved so that the node is not allowed to advertise them as standard CPUs. Instead, the device plugin is used to advertise the 24 isolated cores as separate resources. In this way, standard resources are partitioned into subclasses so that container workloads which require the resources can request them. Without implementing the described techniques, there would be no way for a container workload to be assigned isolated CPU cores as the scheduler does not keep track of isolated CPU cores as a separate resource. In addition, by using a combination of device plugins as well as scheduler plugins, the accounting for resources can be modified as described herein.”; [0167] – “This information (such as the resources/devices to use on the node) can be written by the scheduler plugin into extendible fields such as pod annotations”; [0188]-[0194] – “the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized. […] modify the config file. These modifications can then in tum modify resource/device visibility and CPU visibility as well run any custom resource configuration prior to launching the container. To utilize the device inside the container, the container identifies which resources/devices have been assigned to the container. This can be done via environment variables which are set by both the device plugin as well as the custom runtime configuration modification.”; [0198] – “a resource is assigned to a container ( e.g., using the scheduler plugin extensions).”; [0216] – “Operation 805 illustrates allocating the selected computing resource for use by the requesting workload. In an embodiment, the selected computing resource is allocated for use by the requesting workload in accordance with the set of characteristics associated with the one of the resource classes.”);
instantiate, by a container runtime interface, the container ([0187]-[0193] – “Although the device plugin and scheduling plugin extensions can be used to partition and account for each type of resource independently, a framework to isolate devices and resources can be implemented so that containers cannot use resources which have not been assigned to it. The default container runtime does not have functionality to accomplish this when a container launches. […] In an embodiment, the system is extended by utilizing a custom runtime which intercepts the default runtime when a container is instantiated. At this point, the container configuration is examined and modified so that device and resource visibility is limited. In addition, any resources being used by the container can be configured. Additionally, the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized. […] the following is implemented on a node when a container is instantiated: […] Container orchestration system daemon calls a runtime as a binary which starts the container. […] 3. The CRI then calls the binary runc, passing the configuration file”); and
bind, by the container runtime interface, the container to execute on any of the portion of the plurality of processing devices ([0009] – “The configuration file may be modified based on information provided by the plugins.” ; [0157] – “A custom container runtime interface (CRI) 165, as further discussed below, is used to provide a configuration to ensure that the correct resources are used by containers during container launch 170.”; [0167] – “a process 400 to schedule workloads with simple requests for features on resource type (resource requests can be represented as a count of resources needed with particular feature (for example two CPUs with feature A). […] This information (such as the resources/devices to use on the node) can be written by the scheduler plugin into extendible fields such as pod annotations. […] a custom container runtime interface (CRI), as further discussed below, can be used to provide a configuration to ensure that the correct resources are used by the container.”; [0177] – “this allocation can be written into pod annotations to be used by the custom CRI.”; [0188] – “the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized.”; [0194] – “To utilize the device inside the container, the container identifies which resources/devices have been assigned to the container. This can be done via environment variables which are set by both the device plugin as well as the custom runtime configuration modification.” The environment variables set by the device plugin and CRI indicate the resources (e.g., one or more of the isolated CPU cores of the portion of CPU cores reserved by the device plugin, corresponding to “any of the portion of the plurality of processing devices”) the container being launched is allowed to utilize (“execute on”).).
MEHROTRA fails to expressly teach request, by the orchestrator, allocation by the device plugin.
However, ZHAO teaches request, by the orchestrator, allocation of an amount of processing devices by the device plugin ([0095] – “for example, when remote direct memory access (RDMA) devices are applied in the Kubernetes system, a device plugin mechanism (a mechanism in Kubernetes for device resource management) is often used to facilitate the Kubernetes system to identify and use the RDMA devices”; [0184] – “The VGPU plugin is a device plugin (device plugin) based on kubernetes, which is deployed in all worker nodes of kubernetes”; [0185] – “The VGPU plugin runs on the worker node, communicates with Kubelet (an agent component on the Kubernetes worker node) through a google remote procedure call (GRPC) service”; [0190] – “Kubelet initiates an allocate request to a local VGPU plugin with the VGPU node to be allocated to the container as a parameter. The VGPU plugin receives the allocate request and returns the allocate request to kubelet in the form of an allocate response. Kubelet creates and initializes a container containing specified VPGU resources according to the allocate response.”).
MEHROTRA and ZHAO are considered to be analogous art to the claimed invention because they are reasonably pertinent to the problem faced by the inventor of improving allocation of processing devices to containers using a device plugin. MEHROTRA teaches the device plugin is responsible for managing/assigning (i.e., allocating) isolated CPU cores for use by eligible workloads instead of having the orchestrator manage them, as isolated CPU cores are not typically advertised and scheduled in a typical Kubernetes orchestration framework (see MEHROTRA: [0005], [0007]-[0008], [0157]-[0159]). It also teaches the orchestrator receiving a request for a workload to use resources, where the resources may be isolated CPU cores, and the resources are subsequently allocated for use by the requesting workload (see MEHROTRA: [0156] and [0214]-[0216]). Based on these disclosures of MEHROTRA, it would have been obvious to one of ordinary skill in the art that interaction between the orchestrator receiving the request and the device plugin is necessary in order to achieve allocation of the resources managed by the device plugin for use by the requesting application in the system taught by MEHROTRA, and that using the particular method of interaction between a Kubernetes orchestrator and a device plugin, e.g., the disclosed request, taught by ZHAO to achieve such interaction, where the request is sent from the orchestrator to the device plugin for allocation of the processing resources managed by the device plugin, would yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art to have implemented the request for allocation of resources managed by a device plugin sent from a Kubernetes orchestrator to the device plugin taught by ZHAO in the system of MEHROTRA to achieve allocation of resources managed by the device plugin as requested for a container, and to enable the Kubernetes orchestration system to identify and use the devices managed by the device plugin (see ZHAO: [0093] and [0095]).
Regarding claim 2, the combination of MEHROTRA in view of ZHAO teaches The system of claim 1. MEHROTRA further teaches wherein the device plugin commences execution independent of the orchestrator ([0006]-[0007] – “Plugins can be used to identify and manage available resources […] A number of CPU cores to be isolated are identified, such that when the system boots up, these isolated cores are not available for general use. The orchestration system is notified so that when resources are advertised, the isolated cores are not generally advertised in the same way as other resources. Rather, the isolated cores are advertised as available for eligible workloads. Instead of having the orchestration system advertise isolated cores, device plugins are used to advertise these isolated cores. In an embodiment, the device plug-in is configured to manage and allow eligible workloads to use the isolated cores. The device plug-in excludes the isolated cores from OS scheduling as well as scheduling by the orchestration system.” The cores to be isolated are identified, where identification is performed by the device plugin, at/pre-system boot time (i.e., before the orchestrator is executing on the system) so that the isolated cores are not available for use by the orchestration system, implicitly teaching the device plugin executing before the orchestration system.).
Regarding claim 4, the combination of MEHROTRA in view of ZHAO teaches The system of claim 1. MEHROTRA further teaches wherein:
the container is a first container, the processor request is a first processor request, and the amount is a first amount ([0002] – “techniques for isolating resources from other containers are described along with techniques for configuring devices assigned to the containers.”; [0004] – “Orchestrators set up, monitor, and tear down containers, as well as ensure that hardware resources are assigned to each of the containers.”; [0024] – “managing resource requests from containers to determine workload placement”; [0081]-[0082] – “a request can be made for 2 CPUs with (feature A and (feature B or not feature C)).[…] In addition, one more CPU with just feature A may be requested.”; [0156] – “the workloads are executed by containers 118”; [0157] – “ensure that the correct resources are used by containers during container launch 170”; [0158] – “device plugin is used to advertise the 24 isolated cores as separate resources. In this way, standard resources are partitioned into subclasses so that container workloads which require the resources can request them.”; [0254] – “calls to create the containers and updating a configuration of the containers.”); and
the executable code, when executed by the plurality of processing devices, further causes the plurality of processing devices to:
receive, by the orchestrator, a request for instantiation of a second container including a second processor request for a second amount of processing devices of the device plugin ([0009], [0045], [0096], [0156], [0187]-[0188], [0197], [0214]-[0215] as cited with respect to claim 1. These would be repeated for a second request for a second container requiring isolated CPU cores);
[…] allocation of the second amount by the device plugin ([0007], [0157]-[0159], [0167], [0188]-[0194], [0198], [0216] as cited with respect to claim 1);
instantiate, by the container runtime interface, the second container ([0187]-[0193] as cited with respect to claim 1); and
bind, by the container runtime interface, the second container to execute on any of the portion of the plurality of processing devices ([0009], [0157], [0167], [0177], [0194] as cited with respect to claim 1).
MEHROTRA fails to expressly teach request, by the orchestrator, allocation by the device plugin.
However, ZHAO teaches request, by the orchestrator, allocation of an amount of processing devices by the device plugin ([0095], [0184]-[0185], [0190] as cited with respect to claim 1).
MEHROTRA teaches the device plugin is responsible for managing/assigning (i.e., allocating) isolated CPU cores for use by eligible workloads instead of having the orchestrator manage them, as isolated CPU cores are not typically advertised and scheduled in a typical Kubernetes orchestration framework (see MEHROTRA: [0005], [0007]-[0008], [0157]-[0159]). It also teaches the orchestrator receiving a request for a workload to use resources, where the resources may be isolated CPU cores, and the resources are subsequently allocated for use by the requesting workload (see MEHROTRA: [0156] and [0214]-[0216]). Based on these disclosures of MEHROTRA, it would have been obvious to one of ordinary skill in the art that interaction between the orchestrator receiving the request and the device plugin is necessary in order to achieve allocation of the resources managed by the device plugin for use by the requesting application in the system taught by MEHROTRA, and that using the particular method of interaction between a Kubernetes orchestrator and a device plugin, e.g., the disclosed request, taught by ZHAO to achieve such interaction, where the request is sent from the orchestrator to the device plugin for allocation of the processing resources managed by the device plugin, would yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art to have implemented the request for allocation of resources managed by a device plugin sent from a Kubernetes orchestrator to the device plugin taught by ZHAO in the system of MEHROTRA to achieve allocation of resources managed by the device plugin as requested for a container, and to enable the Kubernetes orchestration system to identify and use the devices managed by the device plugin (see ZHAO: [0093] and [0095]).
Regarding claim 5, the combination of MEHROTRA in view of ZHAO teaches The system of claim 4. MEHROTRA further teaches wherein the executable code, when executed by the plurality of processing devices, further causes the plurality of processing devices to:
allocate, by the device plugin, the first and second amount of the portion to the first container and the second container ([0007] – “the device plug-in is configured to manage and allow eligible workloads to use the isolated cores”; [0045] – “Since each resource is represented separately, the number of each resource that is needed in the container workload can be requested as resource requests. The requested resources can be scheduled onto a node where there are a sufficient number of resources.”; [0157] – “The device plugins 153 advertise other resources which can be used to extend the resources to non-native Kubernetes resources. A custom container runtime interface (CRI) 165, as further discussed below, is used to provide a configuration to ensure that the correct resources are used by containers during container launch 170. Device plugins 153 are used to advertise resource sets and to manage and allow eligible workloads to use the resource sets.”; [0158]-[0159] – “a node may have 32 CPUs, where 24 are isolated CPU cores. Those 24 isolated cores can be reserved so that the node is not allowed to advertise them as standard CPUs. Instead, the device plugin is used to advertise the 24 isolated cores as separate resources. In this way, standard resources are partitioned into subclasses so that container workloads which require the resources can request them. Without implementing the described techniques, there would be no way for a container workload to be assigned isolated CPU cores as the scheduler does not keep track of isolated CPU cores as a separate resource. In addition, by using a combination of device plugins as well as scheduler plugins, the accounting for resources can be modified as described herein.” [0167] – “This information (such as the resources/devices to use on the node) can be written by the scheduler plugin into extendible fields such as pod annotations”; [0188]-[0194] – “the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized. […] modify the config file. These modifications can then in tum modify resource/device visibility and CPU visibility as well run any custom resource configuration prior to launching the container. To utilize the device inside the container, the container identifies which resources/devices have been assigned to the container. This can be done via environment variables which are set by both the device plugin as well as the custom runtime configuration modification.”; [0198] – “a resource is assigned to a container ( e.g., using the scheduler plugin extensions).”; [0216] – “Operation 805 illustrates allocating the selected computing resource for use by the requesting workload. In an embodiment, the selected computing resource is allocated for use by the requesting workload in accordance with the set of characteristics associated with the one of the resource classes.”).
Regarding claim 7, the combination of MEHROTRA in view of ZHAO teaches The system of claim 1. MEHROTRA further teaches wherein:
the portion is a first portion and the container is a first container ([0002] – “techniques for isolating resources from other containers are described along with techniques for configuring devices assigned to the containers.”; [0004] – “Orchestrators set up, monitor, and tear down containers, as well as ensure that hardware resources are assigned to each of the containers.”; [0024] – “managing resource requests from containers to determine workload placement”; [0034] – “cores are classified as isolcpus (CPUs completely isolated from the system) or non-isolated”; [0156] – “the workloads are executed by containers 118”; [0157] – “ensure that the correct resources are used by containers during container launch 170”; [0158] – “For other resources such as CPUs, resources can be removed from the standard advertisement pool and assigned to other resources. For example, a node may have 32 CPUs, where 24 are isolated CPU cores. Those 24 isolated cores can be reserved so that the node is not allowed to advertise them as standard CPUs. Instead, the device plugin is used to advertise the 24 isolated cores as separate resources. In this way, standard resources are partitioned into subclasses so that container workloads which require the resources can request them.”; [0197] – “CPUs in particular can be partitioned into isolcpus and non-isolcpus.”; [0246] – “the computing resources comprise central processing units (CPUs) that are assigned into non-overlapping isolated CPUs and non-isolated CPU resource classes”; [0254] – “calls to create the containers”); and
the executable code, when executed by the plurality of processing devices, further causes the plurality of processing devices to:
receive, by the orchestrator, a request for instantiation of a second container including a second processor request for a second amount of processing devices of the plurality of processing devices, the request for instantiation of the second container not referencing the device plugin ([0009] – “The disclosure also provides a way to launch a given workload […] when a node launches the workload”; [0045] – “the number of each resource that is needed in the container workload can be requested as resource requests. The requested resources can be scheduled onto a node where there are a sufficient number of resources”; [0096] – “The above request can be interpreted as "requests" being an array of resource requests for the workload, with "requests.[i].type" representing the type of resource being requested, and "requests[i].quantity" being the number of these resources being requested. "requests.[i].features" is an array of potential features which qualify as matching the request.”; [0156] – “In response to receiving a request 132B from one of the computing nodes 102 to use one of the resources”; [0187]-[0188] – “the system is extended by utilizing a custom runtime which intercepts the default runtime when a container is instantiated. At this point, the container configuration is examined and modified so that device and resource visibility is limited. In addition, any resources being used by the container can be configured.”; [0197] – “The disclosed embodiments enable partitioning of resources/devices into different classes for resource scheduling purposes using a feature or characteristic. CPUs in particular can be partitioned into isolcpus and non-isolcpus”; [0214] – “each of the resource classes are associated with a set of characteristics. In an embodiment, the characteristics include whether an associated resource is isolated or non-isolated”; [0215] – “in response to receiving, from a workload running in one of the containerized functions, a request to use one of the resource classes, selecting one of the computing resources that is available for use by the requesting workload based on a set of characteristics associated with the one of the resource classes requested by the requesting workload.” When the requested resource class is non-isolated CPUs, the request is “not referencing the device plugin” which manages isolated CPUs, in contrast to the standard Kubernetes orchestration system managing standard resources, e.g., standard (“non-isolated”) CPUs, see [0157]-[0158].);
allocate, by the orchestrator, the second amount of the plurality of processing devices as dedicated to the second container ([0156] – “In response to receiving a request 132B from one of the computing nodes 102 to use one of the resources, the orchestrator reserves the requested resource for use by the computing node 102 that sent the request 132B. The requested resource is reserved for exclusive use for workloads 108 executed by the computing node 102 that sent the request 132B.”; [0216] – “Operation 805 illustrates allocating the selected computing resource for use by the requesting workload. In an embodiment, the selected computing resource is allocated for use by the requesting workload in accordance with the set of characteristics associated with the one of the resource classes.”);
instantiate, by the container runtime interface, the second container ([0187]-[0193] – “the system is extended by utilizing a custom runtime which intercepts the default runtime when a container is instantiated. At this point, the container configuration is examined and modified so that device and resource visibility is limited. In addition, any resources being used by the container can be configured. […] the following is implemented on a node when a container is instantiated: […] Container orchestration system daemon calls a runtime as a binary which starts the container. […] 3. The CRI then calls the binary runc, passing the configuration file”);; and
bind, by the container runtime interface the second container to the second amount of the plurality of processing devices ([0157] – “A custom container runtime interface (CRI) 165, as further discussed below, is used to provide a configuration to ensure that the correct resources are used by containers during container launch 170.”; [0167] – “a custom container runtime interface (CRI), as further discussed below, can be used to provide a configuration to ensure that the correct resources are used by the container.”; [0177] – “this allocation can be written into pod annotations to be used by the custom CRI.”; [0194]-[0195] – “To utilize the device inside the container, the container identifies which resources/devices have been assigned to the container. This can be done via environment variables which are set by both the device plugin as well as the custom runtime configuration modification. The container typically launches a binary which uses a configuration file containing values for various devices to use.”).
Regarding claim 8, the combination of MEHROTRA in view of ZHAO teaches The system of claim 1, MEHROTRA further teaches wherein the orchestrator is a Kubelet ([0190] – “Container orchestration system daemon (e.g., kubelet)”).
Regarding claim 9, the combination of MEHROTRA in view of ZHAO teaches The system of claim 1. MEHROTRA further teaches wherein:
the request references an application type ([0096] – “The above request can be interpreted as "requests" being an array of resource requests for the workload, with "requests.[i].type" representing the type of resource being requested, and "requests[i].quantity" being the number of these resources being requested. ”requests.[i].features" is an array of potential features which qualify as matching the request.”; [0197] – “The disclosed embodiments enable partitioning of resources/devices into different classes for resource scheduling purposes using a feature or characteristic. CPUs in particular can be partitioned into isolcpus and non-isolcpus”; [0214]-[0215] – “each of the resource classes are associated with a set of characteristics. In an embodiment, the characteristics include whether an associated resource is isolated or non-isolated. […] a request to use one of the resource classes, selecting one of the computing resources that is available for use by the requesting workload based on a set of characteristics associated with the one of the resource classes requested by the requesting workload”; [0005] and [0009]-[0010] – a containerized application that needs to be executed on an isolated CPU can be considered a “type” of application, e.g., one that requires low latency or low jitter. Thus, a request which includes a feature/characteristic of a resource/device required by the application to be run in the container, such as a characteristic indicating isolated CPUs, “references an application type”.); and
the executable code, when executed by the plurality of processing devices, further causes the plurality of processing devices to modify the request to reference the device plugin in response to the application type ([0009]-[0010] – “a way to launch a given workload when an isolated resource is being used and to ensure that the isolation is maintained. In an embodiment, when a node launches the workload, the runtime call is intercepted and overridden with a custom binary. The custom binary updates a configuration file that is used to launch a container. The configuration file may be modified based on information provided by the plugins. […] In an embodiment using Kubernetes, webhook mutation can be used by auto injecting an initialization container into the pod. The initialization container can view the workload that is being launched and determine which configuration files need to be modified.”; [0187]-[0188] – “the container configuration is examined and modified so that device and resource visibility is limited. In addition, any resources being used by the container can be configured. Additionally, the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized. Furthermore, instead of writing a full custom runtime, hooks can be added to the existing container runtime.”; [0193]-[0196] – “The CRI then calls the binary rune, passing the configuration file […] A script can be implemented which has the same API as the call to runc, intercept the call to runc, and then modify the config file. These modifications can then in turn modify resource/device visibility and CPU visibility as well run any custom resource configuration prior to launching the container. To utilize the device inside the container, the container identifies which resources/devices have been assigned to the container. This can be done via environment variables which are set by both the device plugin as well as the custom runtime configuration modification. The container typically launches a binary which uses a configuration file containing values for various devices to use. […] Code can be implemented to modify this configuration file using a script. The script can either (1) be a container hook which launches prior to container start, by inserting an init container which has a script to modify the configuration file, or (2) by directly modifying the entry point of the container to launch the script”; [0157]-[0159] – the device and scheduler plugins are used to manage/assign isolated CPUs to container workloads; thus, when the characteristic (reference to an “application type”) of the resource class in the request is indicating an isolated CPU, the configuration file to launch the CPU is modified to indicate isolated CPUs, e.g., as assigned by the plugins).
Regarding claim 10, the combination of MEHROTRA in view of ZHAO teaches The system of claim 9. MEHROTRA further teaches wherein the executable code, when executed by the plurality of processing devices, further causes the plurality of processing devices to modify the request to reference the device plugin in response to the application type using a mutating webhook ([0009]-[0010] – “when a node launches the workload, the runtime call is intercepted and overridden with a custom binary. The custom binary updates a configuration file that is used to launch a container. The configuration file may be modified based on information provided by the plugins.[…] In an embodiment using Kubernetes, webhook mutation can be used by auto injecting an initialization container into the pod. The initialization container can view the workload that is being launched and determine which configuration files need to be modified.”; [0187]-[0188] – “the container configuration is examined and modified so that device and resource visibility is limited. In addition, any resources being used by the container can be configured. Additionally, the container runtime modifications can be used to ensure that the devices assigned by the scheduler plugin are being utilized. Furthermore, instead of writing a full custom runtime, hooks can be added to the existing container runtime.”).
Claims 11-12, 14-15, and 16-20 are directed to A method comprising: the functions performed by the system of claims 1-2, 4-5, and 7-10, respectively. Accordingly, claims 11-12, 14-15, and 16-20 are rejected as being unpatentable over MEHROTRA in view of ZHAO for the same reasons presented with respect to claims 1-2, 4-5, and 7-10 above.
Claim 20 is directed to A non-transitory computer-readable medium storing executable instructions that, when executed by a plurality of processing devices (see MEHROTRA: [0222]-[0224] and [0227]), cause the plurality of processing devices to: perform the functions performed by the system of claim 1. Accordingly, claim 20 is rejected as being unpatentable over MEHROTRA in view of ZHAO for the same reasons presented with respect to claim 1 above.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over MEHROTRA in view of ZHAO as applied to claims 1 and 11 above, and further in view of Moussaoui (U.S. Pub. No. 2022/0357995).
Regarding claim 3, the combination of MEHROTRA in view of ZHAO teaches The system of claim 2, but fails to expressly teach wherein the device plugin is a system daemon.
However, Moussaoui teaches wherein the device plugin is a system daemon ([0098] – “resource management unit may be configured for creating a resource allocation process, for example running in the background such as a daemon process”; [0015] – a “resource management unit” is implemented external to a Kubernetes cluster, that does not require updating the orchestration software environment, i.e., it adds functionality without needing to make changes to the existing container orchestration environment, making it analogous to a “plug-in”; [0026], [0033] and [0096]– the resource management unit allocates system resources, such as CPU resources, to applications at container level in response to requests).
Moussaoui is considered to be analogous art to the claimed invention because it is reasonably pertinent to the problem faced by the inventor of efficiently allocating processing devices to containers. Therefore, it would have been obvious to one of ordinary skill in the art to have modified the device plugin of MEHROTRA in view of ZHAO, such that it is implemented a system daemon as taught by Moussaoui. Doing so would enable monitoring and identification of the current status of CPU resource allocations to the containers, which could allow for dynamic modification of the allocation of CPU resources (see Moussaoui: [0153]-[0154]).
Claim 13 recites substantially the same additional limitations as claim 3, applied to the method of claim 12. Accordingly, claim 13 is rejected as being unpatentable over MEHROTRA in view of ZHAO and further in view of Moussaoui for the same reasons presented with respect to claim 3.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over MEHROTRA in view of ZHAO as applied to claim 4 above, and further in view of Krasner et al. (U.S. Pub. No. 2021/0373956), hereinafter Krasner.
Regarding claim 6, the combination of MEHROTRA in view of ZHAO teaches The system of claim 4, but fails to expressly teach wherein the first amount and the second amount are fractional.
However, Krasner teaches wherein the first amount and the second amount are fractional ([0020] – “Each computing module may include one or more central processing units (CPUs). In at least one embodiment, each of the CPUs may be a multi-core CPU including multiple processing cores or processors. The individual cores or processors within a single CPU”. CPU cores may be fractions (i.e., portions) of a single processor (e.g., of multi-core processor; [0067] – “The container hypervisors 531, 532 map virtual CPU cores to the physical CPU cores 511, 512 of the hardware layer 510. That is, in accordance with the system described herein, the one or more Guest OS's may only have access to a different number of available CPU cores (virtual CPU cores) that are available as physical CPU cores on the hardware 510. Through the use of the container hypervisors 531, 532 running as storage system OS threads t0, t1, and t2 (rather than being scheduled as storage system OS instances), at least one embodiment of the system described herein provides for the ability to schedule fractions of CPU time on multiple cores”).
Krasner is considered to be analogous art to the claimed invention because it is reasonably pertinent to the problem faced by the inventor of efficiently allocating processing devices to containers. Therefore, it would have been obvious to one of ordinary skill in the art that first amount and second amount of resources, e.g., amounts of CPU cores, requested and allocated to containers as taught by MEHROTRA in view of ZHAO could be fractional as taught by Krasner, e.g., as a fraction of cores of a multi-core processor or as fractions of CPU time on the cores. Incorporating the resource allocation methods of Krasner, including allocation of fractional amounts of processing resources, would enable dynamic allocation of CPU resources to threads of the multiple containerized workloads according to the demands of the workloads at different times while ensuring each application meets its target service level and avoiding resource waste (see Krasner: [0069]-[0071]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL Reference “Plug-in (computing)” (see attached PTO-892) teaches a plug-in is a software component that adds a feature to an existing computer program, e.g., without needing to make changes to the existing computer program (see Pages 1-2).
Tang et al. (U.S. Pub. No. 2022/0283869) teaches a kubelet is configured to invoke an Allocate interface of a device plugin (see [0039]-[0040]), and that allocated resources are bound to a pod object (see [0041]).
Sharma et al. (U.S. Pub. No. 2024/0422107) teaches an orchestrator uses a container runtime interface to set the CPUs on which containers can run, where some CPUs may be reserved and excluded from an allocation pool managed by a CPU manager, where the shared pool that remains is the set of CPUs on which any containers in BestEffort and Burstable pods run (see [0150]-[0153]).
Shan et al. (U.S. Pub. No. 2015/0040131) teaches creating an advanced resource pool and a default resource pool by dividing physical cores of a CPU according to service levels of the pools, where the cores allocatable to virtual machines (see Abstract).
GUO (CN 115858083 A) teaches an agent module (i.e., a kubelet) dividing CPU resources on a node into a plurality of different resource pools, receiving a container creation request includes a resource pool label and resource demand, invoking a container runtime interface (CRI) to create the container, determining an index CPU core corresponding to the resource pool label, and sending the index CPU core and resource pool label to the CRI so that the CRI binds the container and resource pool corresponding to the CPU core (see entire document).
Asveren et al. (U.S. Pub. No. 2021/0328858) teaches a CNI plugin operating independently of a Kubelet (see [0054]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER MARIE GUTMAN whose telephone number is (703)756-1572. The examiner can normally be reached M-F: 8:00 am - 4:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kevin Young can be reached at 571-270-3180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNIFER MARIE GUTMAN/Examiner, Art Unit 2194 /KEVIN L YOUNG/Supervisory Patent Examiner, Art Unit 2194
