Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-15 are presented for examination.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Use of the word “means” (or “step for”) in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function.
Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim 13 elements in this application that use the word “application scheduler" and “a migration controller” and “placement controller” and “a resource synchronizer” has/have been interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because it uses/they use a generic placeholder “means for ” coupled with functional language “storing” without reciting sufficient structure to achieve the function. Furthermore, the generic placeholder is not preceded by a structural modifier.
Further details about lack of structure are provided below as part of 112 (b) rejection and will not be repeated here.
If applicant wishes to provide further explanation or dispute the examiner’s interpretation of the corresponding structure (or lack of it), applicant must identify the corresponding structure with reference to the specification by page and line number, and to the drawing, if any, by reference characters in response to this Office action.
If applicant does not intend to have the claim limitation(s) treated under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112 , sixth paragraph, applicant may amend the claim(s) so that it/they will clearly not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, or present a sufficient showing that the claim recites/recite sufficient structure, material, or acts for performing the claimed function to preclude application of 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
For more information, see MPEP § 2173 et seq. and Supplementary Examination Guidelines for Determining Compliance With 35 U.S.C. 112 and for Treatment of Related Issues in Patent Applications, 76 FR 7162, 7167 (Feb. 9, 2011).
“means” (or “step for”) are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word “means” (or “step for”) are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 13 and 14 are rejected under 35 U.S.C. 112(a), as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
As per claim 13 the limitations reciting “ application scheduler" and “a migration controller” and “placement controller” and “a resource synchronizer”” lack adequate structure in the specification showing possession of the invention.
The remaining claims, not specifically mentioned, are rejected for being dependent upon claim 13.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-15 are rejected under 35 U.S.C. 112(b), as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
In claim 1, the term “selecting, when a workload which is impossible to distribute due to lack of resources is detected in a logical cloud including a plurality of clusters which are present in multiple clouds, one or more first workloads to be migrated from a first cluster to another cluster by referring to a predetermined policy, and a workload intent set for workloads executed in the logical cloud”. is confusing. Do we select a workload (based on policy and intent) and then realize it is not possible to migrate it (due to lack of resources) or is “impossibility” itself a criteria?. The examiner will take policy and intent to be the criteria and fact that it cannot be migrated to come after selection. The entire claim is structured around determining a workload needs to migrated, followed by deleting another workload from the destination ( to make room for the migrated workload) and then migrating the workload. This is supported by what is disclosed in the specification ([0026] distribute the workload which is impossible to distribute to the first cluster after one or more first workloads are deleted)
This makes the following part of the claim inconsistent “migrating the one or more first workloads to the second cluster, and deleting the one or more first workloads from the first cluster”. One would expect deleting to happen before migration but the claim has it backwards.
Claims 13 and 15 have the same problem and are rejected for the same reasons.
The remaining claims, not specifically mentioned, are rejected for being dependent upon one of the claims above.
Claim 5, discloses “before step (a) above, selecting a cluster X set in which resources remain and a cluster Y set in which there is no resource, but the node scaling is possible among a plurality of clusters designated as a distribution target of the workload; and trying the distribution of the workload in order from the cluster X set to the cluster Y set”.
The examiner does not understand how cluster Y which has no resource can even be a candidate for “node scaling”. Examiner will take this to be node scaling based on a decreasing order of resource availability among different clusters.
Claim 6, discloses “wherein steps (a) to (d) above are performed when the workload distribution is unsuccessful in both the cluster X set and the cluster Y set”
This seems to imply (based on steps of claim 1) that one needs to migrate when autoscaling of nodes is not possible. Examiner (in the absence of further guidance from the specification) will take this to be rebalancing (migration) in case of node failure or a failure to scale the resources of nodes.
As per claim 13, the limitations reciting “ application scheduler" and “a migration controller” and “placement controller” and “a resource synchronizer” are limitations that invokes 35 U.S.C. 112, sixth paragraph.
However, the written description fails to disclose the corresponding structure, material, or acts for the claimed function.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112, sixth paragraph; or
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the claimed function without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim 14 is rejected for being dependent upon claim 13.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 13-15 rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1).
As per claim 1, Lubsey teaches A method for managing Kubernetes cluster resources in a multi-cloud environment, the method comprising:
(a) selecting, when a workload which is impossible to distribute due to lack of resources is detected in a logical cloud including a plurality of clusters which are present in multiple clouds, one or more first workloads to be migrated from a first cluster to another cluster by referring to a predetermined policy, and a workload intent set for workloads executed in the logical cloud;
(Lubsey [claim 11] wherein migrating the one or more virtual machines executing the one or more workloads of the customer comprises providing room for a busy service within one or more of the plurality of clusters from which the one or more virtual machines executing the one or more workloads of the customer are to be migrated and [col 8, lines 30-41] the Cgroups for performance information relating to the group and the VMs running within it. Further, it may be configured to initiate a migration of a VM based on its workload to an alternative Cgroup. This may be done to provide room for a "busy" service within its originating Cgroup. For example, if the quantity of pools at high water mark is less than the total quantity of pools [policy used for migration], then functions such as defining free space in lower watermark pools, defining "best fit VMs" and vMotion may be performed. Here, vMotion may be the migration of a VM from one host 40 to another while the machine continues to run and accept new requests [accept migration from the workload that was impossible to migrate and [col 5, lines 36-46] FIG. 3 is a diagram illustrating exemplary aspects of resource management service provision according to a disclosed implementation. Here, for example, resources in the data center may be managed so that VMs/workloads running on hosts with a guaranteed service [workload intent] level can be supported on other hosts in the event of a failure. Referring to FIG. 3, in the event of a failure on hosts A and B, a guaranteed workload 314a can be supported via migration to resource location(s) 314b on the remaining hosts C and D [workload intent affects migration] Note that the term “resource units” is also referred to herein as “infrastructure units” (IU) or data center units and [col 5, lines 27-28] In this case, the 10 uVM workload would not meet service level requirement [workload intent] and thus needs to be placed on a different cluster)
(d) distributing the workload which is impossible to distribute to the first cluster. (Lubsey [claim 11] wherein migrating the one or more virtual machines executing the one or more workloads of the customer comprises providing room for a busy service within one or more of the plurality of clusters from which the one or more virtual machines executing the one or more workloads of the customer are to be migrated)
Lubsey does not teach (b) selecting a second cluster to which the one or more first workloads are to be migrated; (c) migrating the one or more first workloads to the second cluster, and deleting the one or more first workloads from the first cluster.
However, Gavali teaches (b) selecting a second cluster to which the one or more first workloads are to be migrated; (c) migrating the one or more first workloads to the second cluster, and deleting the one or more first workloads from the first cluster; (Gavali [0052] If the average resource utilization of a worker node is below the lower threshold, then again trigger workload redistribution. Select worker nodes that have higher utilization (i.e., not only over-utilized worker nodes). Select the workloads in these higher utilization worker nodes for migration so that once these selected workloads are added to the under-utilized worker nodes [migration to another cluster], the total resource utilization of these under-utilized worker nodes will raise above the lower threshold. Delete the selected workloads from the higher utilization worker nodes)
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Gavali with the system of Lubsey to migrate to another cluster. One having ordinary skill in the art would have been motivated to use Gavali into the system of Lubsey for the purpose of rescheduling workloads across worker nodes of a workload orchestration environment to redistribute the workloads based on policy (Gavali paragraph 01)
In Gavali, under and over utilized nodes are grouped as being hot (over) and cold (under) utilized. This is shown in the following paragraph ([0037] Further, worker node group 230 may include hot region 236 and cold region 238 based on metrics 234 exceeding thresholds 240. Thresholds 240 include overutilization threshold 242 and underutilization threshold 244. If metrics 234 of a particular worker node exceed overutilization threshold 242, then workload orchestration manager may place that particular worker node in hot region 236. Similarly, if metrics 234 of a particular worker node exceed underutilization threshold 244, then workload orchestration manager may place that particular worker node in cold region 238..).
The examiner will take this “clustering” to be between hot and cold group of nodes. This is quite consistent with the concept of policy in Lubsey. Gavali explicitly teaches this usage of policy in placing of the workload ([0066] Illustrative embodiments perform workload redistribution to ensure that hot and cold regions are normalized. This normalization is achieved by selecting workloads from the hot region and relocating them in the cold region. Illustrative embodiments perform this normalization process iteratively while ensuring that the constraints, such as, for example, workload placement policy, workload node affinity, and the like, are not broken).
As per claim 2, Gavali teaches The method of claim 1, further comprising: before step (a) above, configuring one or more logical clouds including the plurality of clusters through a management cluster; and monitoring a resource of each of the plurality of clusters through a cluster application programming interface (API). (Gavali 0035] Workload orchestration manager 218 controls the process of rescheduling workloads across worker nodes of a workload orchestration environment to redistribute the workloads based on policy. In this example, workload orchestration manager 218 includes objective function 220, distance function 222, and grouping algorithm 224. Workload orchestration manager 218 utilizes objective function 220 to measure performance of worker nodes and groups of worker nodes based on collected worker node metrics [logical cloud] and optimize the performance of the worker nodes and groups of worker nodes. Workload orchestration manager 218 utilizes distance function 222 to determine similarities and functional relationships between worker nodes and groups of worker nodes. Workload orchestration manager 218 utilizes grouping algorithm 224 to cluster worker nodes into groups of worker nodes based on values provided by objective function 220 and distance function 222. Grouping algorithm 224 may be, for example, a k-means clustering algorithm or the like.
[0037] Workload orchestration manager 218 collects metrics 234 for each worker node in worker nodes 232. Metrics 234 may include, for example, processor utilization metrics, memory utilization metrics, storage utilization metrics, network utilization metrics, health index, and the like, corresponding to each worker node. Further, worker node group 230 may include hot region 236 and cold region 238 based on metrics 234 exceeding thresholds 240. Thresholds 240 include overutilization threshold 242 and underutilization threshold 244. If metrics 234 of a particular worker node exceed overutilization threshold 242, then workload orchestration manager may place that particular worker node in hot region 236. Similarly, if metrics 234 of a particular worker node exceed underutilization threshold 244, then workload orchestration manager may place that particular worker node in cold region 238 [0070] In this example, workload orchestration environment 300 comprises cluster of worker node groups 302. Cluster of worker node groups 302 includes worker node group 304. In this example, worker node group 304 includes master node 306, worker node 1 308, worker node 2 310, and worker node 3 312. However, it should be noted that worker node group 304 may include any number of worker nodes. In addition, master node 306 is the initial entry and management point for all workload operations to be executed in worker node group 304).
As per claim 3, Gavali teaches wherein step (a) above includes:
invoking, by an application scheduler, a migration controller when the workload which is impossible to distribute is detected; selecting, by the migration controller, one or more first workloads to be migrated to another cluster by referring to the policy and the workload intent; and selecting, by a placement controller, the second cluster to which the one or more first workloads are to be migrated according to the invoking by the application scheduler. (Gavali [0048] A cloud manager may provide user visibility, application-centric management (e.g., policies, deployments, health, and operations), and policy-based compliance across clouds and clusters of worker node groups. The cloud manager may provide control of the clusters of worker node groups in the workload orchestration environment. The cloud manager may ensure that a customer's cluster of worker node groups is secure, operating efficiently, and delivering service levels that applications expect. In addition, the cloud manager may deploy workloads on multiple groups of worker nodes. A user may view the workloads in a namespace in all development groups. [0052] If the average resource utilization of a worker node is below the lower threshold, then again trigger workload redistribution. Select worker nodes that have higher utilization (i.e., not only over-utilized worker nodes). Select the workloads in these higher utilization worker nodes for migration so that once these selected workloads are added to the under-utilized worker nodes [migration to another cluster], the total resource utilization of these under-utilized worker nodes will raise above the lower threshold. Delete the selected workloads from the higher utilization worker nodes)
As per claim 13, Lubsey teaches A system for managing a cluster resource in multiple clouds, the system comprising:
an application scheduler detecting a workload which is impossible to distribute due to lack of resources in a logical cloud including a plurality of clusters which are present in the multiple clouds; (Lubsey [col 2, lines 5-15] Here, for example, the implementations of FIGS. 1A-1C and elsewhere herein may be implemented in a cloud computing arrangement, i.e., with the various clusters physically present at differing/distributed locations throughout the cloud. Referring to FIG. 1A, a system 100 [scheduler] of managing resources of customers 102 across a variety of clusters 104 is shown. As set forth in more detail elsewhere, a determination of the resource needs of customers may be performed, yielding a specified quantity of infrastructure units (IU s) for each customer and [claim 11] wherein migrating the one or more virtual machines executing the one or more workloads of the customer comprises providing room for a busy service within one or more of the plurality of clusters from which the one or more virtual machines executing the one or more workloads of the customer are to be migrated)
a migration controller selecting one or more first workloads to be migrated to another cluster from a first cluster by referring to a predetermined policy, and a workload intent set for workloads which are executed in the logical cloud, by an invoking by the application scheduler; a placement controller selecting a second cluster to which the one or more first workloads are to be migrated according to the invoking by the application scheduler; (Lubsey [col 8, lines 17-24] A management service (or process or application), such as 114 in FIG. 1A, may be provided for monitoring and management of the workloads within the cluster. This management service may be configured in a number of ways. For example, such management service can be configured to respond to highwatermark “alerts” from the Cgroups (control groups) and/or process migration instructions related thereto. and [col 8, lines 30-41] the Cgroups for performance information relating to the group and the VMs running within it. Further, it may be configured to initiate a migration of a VM based on its workload to an alternative Cgroup. This may be done to provide room for a "busy" service within its originating Cgroup. For example, if the quantity of pools at high water mark is less than the total quantity of pools [policy used for migration], then functions such as defining free space in lower watermark pools, defining "best fit VMs" and vMotion may be performed. Here, vMotion may be the migration of a VM from one host 40 to another while the machine continues to run and accept new requests [accept migration from the workload that was impossible to migrate and [col 5, lines 36-46] FIG. 3 is a diagram illustrating exemplary aspects of resource management service provision according to a disclosed implementation. Here, for example, resources in the data center may be managed so that VMs/workloads running on hosts with a guaranteed service [workload intent] level can be supported on other hosts in the event of a failure. Referring to FIG. 3, in the event of a failure on hosts A and B, a guaranteed workload 314a can be supported via migration to resource location(s) 314b on the remaining hosts C and D [workload intent affects migration] Note that the term “resource units” is also referred to herein as “infrastructure units” (IU) or data center units and [col 5, lines 27-28] In this case, the 10 uVM workload would not meet service level requirement [workload intent] and thus needs to be placed on a different cluster)
Lubsey does not teach a resource synchronizer migrating the one or more first workloads to the second cluster by referring to AppContext in which information on the one or more first workloads and the second cluster is updated, deleting the one or more first workloads from the first cluster, and distributing the workload which is impossible to distribute to the first cluster after the one or more first workloads are deleted.
However, Gavani teaches a resource synchronizer migrating the one or more first workloads to the second cluster by referring to AppContext in which information on the one or more first workloads and the second cluster is updated, deleting the one or more first workloads from the first cluster, and distributing the workload which is impossible to distribute to the first cluster after the one or more first workloads are deleted. (Gavali [0073] Workload reschedule 334 [resource synchronizer] fetches new metrics, error messages, and the like from metrics collector 332 at regular intervals. Workload reschedule 334 also determines unhealthy worker nodes within worker node group 304 and migrates the workloads of the unhealthy worker nodes to healthy worker nodes within worker node group 304 [0052] If the average resource utilization of a worker node is below the lower threshold, then again trigger workload redistribution. Select worker nodes that have higher utilization (i.e., not only over-utilized worker nodes). Select the workloads in these higher utilization worker nodes for migration so that once these selected workloads are added to the under-utilized worker nodes [migration to another cluster], the total resource utilization of these under-utilized worker nodes will raise above the lower threshold. Delete the selected workloads from the higher utilization worker nodes)
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Gavali with the system of Lubsey to migrate to another cluster. One having ordinary skill in the art would have been motivated to use Gavali into the system of Lubsey for the purpose of rescheduling workloads across worker nodes of a workload orchestration environment to redistribute the workloads based on policy (Gavali paragraph 01)
In Gavali, under and over utilized nodes are grouped as being hot (over) and cold (under) utilized. This is shown in the following paragraph ([0037] Further, worker node group 230 may include hot region 236 and cold region 238 based on metrics 234 exceeding thresholds 240. Thresholds 240 include overutilization threshold 242 and underutilization threshold 244. If metrics 234 of a particular worker node exceed overutilization threshold 242, then workload orchestration manager may place that particular worker node in hot region 236. Similarly, if metrics 234 of a particular worker node exceed underutilization threshold 244, then workload orchestration manager may place that particular worker node in cold region 238..).
The examiner will take this “clustering” to be between hot and cold group of nodes. This is quite consistent with the concept of policy in Lubsey. Gavali explicitly teaches this usage of policy in placing of the workload ([0066] Illustrative embodiments perform workload redistribution to ensure that hot and cold regions are normalized. This normalization is achieved by selecting workloads from the hot region and relocating them in the cold region. Illustrative embodiments perform this normalization process iteratively while ensuring that the constraints, such as, for example, workload placement policy, workload node affinity, and the like, are not broken).
As per claim 14, Gavali teaches wherein the application scheduler, the migration controller, the placement controller, and the resource synchronizer are managed through a management cluster, and the management cluster further includes a cluster API configuring one or more logical clouds including a plurality of clusters, and monitoring resources of the plurality of respective clusters. (Gavali 0035] Workload orchestration manager 218 controls the process of rescheduling workloads across worker nodes of a workload orchestration environment to redistribute the workloads based on policy. In this example, workload orchestration manager 218 includes objective function 220, distance function 222, and grouping algorithm 224. Workload orchestration manager 218 utilizes objective function 220 to measure performance of worker nodes and groups of worker nodes based on collected worker node metrics [logical cloud] and optimize the performance of the worker nodes and groups of worker nodes. Workload orchestration manager 218 utilizes distance function 222 to determine similarities and functional relationships between worker nodes and groups of worker nodes. Workload orchestration manager 218 utilizes grouping algorithm 224 to cluster worker nodes into groups of worker nodes based on values provided by objective function 220 and distance function 222. Grouping algorithm 224 may be, for example, a k-means clustering algorithm or the like. [0037] Workload orchestration manager 218 collects metrics 234 for each worker node in worker nodes 232. Metrics 234 may include, for example, processor utilization metrics, memory utilization metrics, storage utilization metrics, network utilization metrics, health index, and the like, corresponding to each worker node. Further, worker node group 230 may include hot region 236 and cold region 238 based on metrics 234 exceeding thresholds 240. Thresholds 240 include overutilization threshold 242 and underutilization threshold 244. If metrics 234 of a particular worker node exceed overutilization threshold 242, then workload orchestration manager may place that particular worker node in hot region 236. Similarly, if metrics 234 of a particular worker node exceed underutilization threshold 244, then workload orchestration manager may place that particular worker node in cold region 238 [0070] In this example, workload orchestration environment 300 comprises cluster of worker node groups 302. Cluster of worker node groups 302 includes worker node group 304. In this example, worker node group 304 includes master node 306, worker node 1 308, worker node 2 310, and worker node 3 312. However, it should be noted that worker node group 304 may include any number of worker nodes. In addition, master node 306 is the initial entry and management point for all workload operations to be executed in worker node group 304).
As per claim 15, Lubsey teaches An apparatus for managing a cluster resource in multiple clouds, the apparatus comprising: a processor; and a memory connected to the processor, (Lubsey [col 12, lines 43-48] Each module may even be implemented as a software program stored on a tangible memory (e.g., random access memory, read only memory, CD-ROM memory, hard disk drive, etc.) to be read by a central processing unit to implement the functions of the innovations herein)
wherein the memory stores program instructions executed by the processor to select, when a workload which is impossible to distribute due to lack of resources is detected in a logical cloud including a plurality of clusters which are present in multiple clouds, one or more first workloads to be migrated from a first cluster to another cluster by referring to a predetermined policy, and a workload intent set for workloads executed in the logical cloud (Lubsey [claim 11] wherein migrating the one or more virtual machines executing the one or more workloads of the customer comprises providing room for a busy service within one or more of the plurality of clusters from which the one or more virtual machines executing the one or more workloads of the customer are to be migrated and [col 8, lines 30-41] the Cgroups for performance information relating to the group and the VMs running within it. Further, it may be configured to initiate a migration of a VM based on its workload to an alternative Cgroup. This may be done to provide room for a "busy" service within its originating Cgroup. For example, if the quantity of pools at high water mark is less than the total quantity of pools [policy used for migration], then functions such as defining free space in lower watermark pools, defining "best fit VMs" and vMotion may be performed. Here, vMotion may be the migration of a VM from one host 40 to another while the machine continues to run and accept new requests [accept migration from the workload that was impossible to migrate and [col 5, lines 36-46] FIG. 3 is a diagram illustrating exemplary aspects of resource management service provision according to a disclosed implementation. Here, for example, resources in the data center may be managed so that VMs/workloads running on hosts with a guaranteed service [workload intent] level can be supported on other hosts in the event of a failure. Referring to FIG. 3, in the event of a failure on hosts A and B, a guaranteed workload 314a can be supported via migration to resource location(s) 314b on the remaining hosts C and D [workload intent affects migration] Note that the term “resource units” is also referred to herein as “infrastructure units” (IU) or data center units and
[col 5, lines 27-28] In this case, the 10 uVM workload would not meet service level requirement [workload intent] and thus needs to be placed on a different cluster)
Lubsey does not teach select a second cluster to which the one or more first workloads are to be migrated, migrate the one or more first workloads to the second cluster, delete the one or more first workloads from the first cluster, and distribute the workload which is impossible to distribute to the first cluster after the one or more first workloads are deleted.
However, Gavali teaches select a second cluster to which the one or more first workloads are to be migrated, migrate the one or more first workloads to the second cluster, delete the one or more first workloads from the first cluster, and distribute the workload which is impossible to distribute to the first cluster after the one or more first workloads are deleted. (Gavali [0052] If the average resource utilization of a worker node is below the lower threshold, then again trigger workload redistribution. Select worker nodes that have higher utilization (i.e., not only over-utilized worker nodes). Select the workloads in these higher utilization worker nodes for migration so that once these selected workloads are added to the under-utilized worker nodes [migration to another cluster], the total resource utilization of these under-utilized worker nodes will raise above the lower threshold. Delete the selected workloads from the higher utilization worker nodes)
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Gavali with the system of Lubsey to migrate to another cluster. One having ordinary skill in the art would have been motivated to use Gavali into the system of Lubsey for the purpose of rescheduling workloads across worker nodes of a workload orchestration environment to redistribute the workloads based on policy (Gavali paragraph 01)
In Gavali, under and over utilized nodes are grouped as being hot (over) and cold (under) utilized. This is shown in the following paragraph ([0037] Further, worker node group 230 may include hot region 236 and cold region 238 based on metrics 234 exceeding thresholds 240. Thresholds 240 include overutilization threshold 242 and underutilization threshold 244. If metrics 234 of a particular worker node exceed overutilization threshold 242, then workload orchestration manager may place that particular worker node in hot region 236. Similarly, if metrics 234 of a particular worker node exceed underutilization threshold 244, then workload orchestration manager may place that particular worker node in cold region 238..).
The examiner will take this “clustering” to be between hot and cold group of nodes. This is quite consistent with the concept of policy in Lubsey. Gavali explicitly teaches this usage of policy in placing of the workload ([0066] Illustrative embodiments perform workload redistribution to ensure that hot and cold regions are normalized. This normalization is achieved by selecting workloads from the hot region and relocating them in the cold region. Illustrative embodiments perform this normalization process iteratively while ensuring that the constraints, such as, for example, workload placement policy, workload node affinity, and the like, are not broken).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Navali (US 2022/0353201 A1).
As per claim 4, Lubsey and Gavali do not teach wherein the cluster API stores information on whether node scaling is possible in each of the plurality of clusters.
However, Navali teaches wherein the cluster API stores information on whether node scaling is possible in each of the plurality of clusters. (Navali [0021] In some arrangements, the cluster includes a plurality of compute nodes and a node scaling circuit. Additionally, changing the number of compute nodes allocated to the application by the cluster includes providing a signal to the node scaling circuit that directs the node scaling circuit to increase a first number of compute nodes allocated to the application by the cluster to a second number of compute nodes allocated to the application by the cluster to proactively address the predicted change in demand on the application. Similarly, in some arrangements, a signal may be provided to the node scaling circuit that directs the node scaling circuit to decrease the number of compute nodes to proactively address predicted changes in demand on the application (e.g., to reduce or de-allocate resources when predicted demand is low)).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Navali with the system of Lubsey and Gavali to store node scaling information. One having ordinary skill in the art would have been motivated to use Navali into the system of Lubsey and Gavali for the purpose of implementing improved techniques directed to controlling placement of workloads of an application within an application environment by predicting a future change in demand. (Navali paragraph 08)
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Navali (US 2022/0353201 A1) and Kim (US 2022/0329651 A1).
As per claim 5, Lubsey, Gavali and Navali do not teach before step (a) above, selecting a cluster X set in which resources remain and a cluster Y set in which there is no resource, but the node scaling is possible among a plurality of clusters designated as a distribution target of the workload; and trying the distribution of the workload in order from the cluster X set to the cluster Y set.
However, Kim teaches before step (a) above, selecting a cluster X set in which resources remain and a cluster Y set in which there is no resource, but the node scaling is possible among a plurality of clusters designated as a distribution target of the workload; and trying the distribution of the workload in order from the cluster X set to the cluster Y set. (Kim [0021] Here, performing the service node scaling may be configured such that, when cluster resource utilization, measured based on the resource utilization of each service node, falls out of a preset target range of cluster resource utilization, the service node is added or deleted in consideration of the resource utilization of each service node and network proximity between a cloud region and a group of devices using service, which are grouped in consideration of the rate of increase of access and the frequency of access. [0022] Here, performing the service node scaling may include, when the cluster resource utilization is greater than an upper limit of the preset target range of the cluster resource utilization, selecting a target cloud region, in which a new service node is to be added, from among a first cloud region, including a service node having resource utilization exceeding an upper limit of the preset target range of the service node resource utilization, among multiple service nodes constituting the container orchestration cluster, and a second cloud region, selected in consideration of network proximity between the cloud region and the group of devices using service. [0130] Also, when it is determined at step S635 that the resource utilization of the cluster is greater than the preset target lower limit, the process of service node scaling, which starts from step S610, is repeatedly performed, whereby continuous management may be performed to balance the service node execution load).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Kim with the system of Lubsey and Gavali and Navali to scale based on a decreasing order of resource availability. One having ordinary skill in the art would have been motivated to use Kim into the system of Lubsey and Gavali and Navali for the purpose of implementing container orchestration in an environment of multiple geographically distributed clouds. (Kim paragraph 02)
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Navali (US 2022/0353201 A1) and Kim (US 2022/0329651 A1) and Capano (US 2025/0348399 A1).
As per claim 6, Lubsey, Gavali and Navali and Kim do not teach wherein steps (a) to (d) above are performed when the workload distribution is unsuccessful in both the cluster X set and the cluster Y set.
However, Capano teaches wherein steps (a) to (d) above are performed when the workload distribution is unsuccessful in both the cluster X set and the cluster Y set. (Capano [0154] In one or more embodiments, the deallocating the one or more resources includes terminating low-priority services or migrating workloads to lower-utilization hardware. [0178] In one or more embodiments, these operational modifications may be coordinated through integration with orchestration platforms (e.g., Kubernetes, Apache Mesos, AWS Auto Scaling) and may include application-specific responses such as scaling out, throttling, rebooting, or redeploying services. [0206] In one or more embodiments, the controller communicates with an infrastructure orchestration layer using standard APIs to provision, resize, or remove computing resources. For example, the system may use Kubernetes Horizontal Pod Autoscaler to increase the number of application containers in response to predicted CPU saturation. [0208] In at least one embodiment, the controller may communicate with orchestration or resource management frameworks via API calls or control interfaces. For example, it may instruct a Kubernetes cluster to scale application pods, initiate the creation of new virtual machines via a cloud provider API, or terminate idle containers using local container management commands. [0218] For example, if the model predicts sustained CPU overload beyond a safe threshold, the controller may initiate resource reallocation commands to a redundant CPU cluster or reassign workloads through a load balancer. Likewise, when resource utilization is predicted to remain low, the controller may temporarily scale down resources or containerized services to reduce energy and cost overhead).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Capano with the system of Lubsey and Gavali and Navali and Kim to scale based on a decreasing order of resource availability. One having ordinary skill in the art would have been motivated to use Capano into the system of Lubsey and Gavali and Navali and Kim for the purpose of implementing an iterative method, a device and a system for monitoring a computing device (Capano paragraph 02)
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Capano (US 2025/0348399 A1).
As per claim 7, Lubsey and Gavali do not teach wherein when there is a scale-out request of an application which is being currently executed, steps (a) to (d) above are performed when resources of the clusters receiving the scale-out request are insufficient, and node scaling in the clusters receiving the scale-out request is impossible.
However, Capano teaches wherein when there is a scale-out request of an application which is being currently executed, steps (a) to (d) above are performed when resources of the clusters receiving the scale-out request are insufficient, and node scaling in the clusters receiving the scale-out request is impossible. (Capano [0154] In one or more embodiments, the deallocating the one or more resources includes terminating low-priority services or migrating workloads to lower-utilization hardware. [0178] In one or more embodiments, these operational modifications may be coordinated through integration with orchestration platforms (e.g., Kubernetes, Apache Mesos, AWS Auto Scaling) and may include application-specific responses such as scaling out, throttling, rebooting, or redeploying services. [0206] In one or more embodiments, the controller communicates with an infrastructure orchestration layer using standard APIs to provision, resize, or remove computing resources. For example, the system may use Kubernetes Horizontal Pod Autoscaler to increase the number of application containers in response to predicted CPU saturation. [0208] In at least one embodiment, the controller may communicate with orchestration or resource management frameworks via API calls or control interfaces. For example, it may instruct a Kubernetes cluster to scale application pods, initiate the creation of new virtual machines via a cloud provider API, or terminate idle containers using local container management commands. [0218] For example, if the model predicts sustained CPU overload beyond a safe threshold, the controller may initiate resource reallocation commands to a redundant CPU cluster or reassign workloads through a load balancer. Likewise, when resource utilization is predicted to remain low, the controller may temporarily scale down resources or containerized services to reduce energy and cost overhead).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Capano with the system of Lubsey and Gavali to act when resources are insufficient. One having ordinary skill in the art would have been motivated to use Capano into the system of Lubsey and Gavali for the purpose of iterative method, a device and a system for monitoring a computing device (Capano paragraph 02)
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Pan (US 2021/0019160 A1).
As per claim 8, Lubsey and Gavali do not teach wherein the workload intent includes whether it is possible to migrate each workload and priority information.
However, Pan teaches wherein the workload intent includes whether it is possible to migrate each workload and priority information. (Pan [0050] In one example, the priority placement primitive 414 applies to group power-on of several workloads during initial placement 452. But the priority placement primitive 414 can also apply when a group of workloads should be evacuated from the host as part of the host maintenance workflow 454. When a host is put into maintenance mode, by default the workloads can be migrated away from the host. However, the migration can be prioritized based on the priority placement primitive 414. This can include migrating the higher-priority workloads prior to the lower-priority workloads. Additionally, at the new host, the priority placement primitive 414 can again be used as part of the initial placement workflow 452, in an example).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Pan with the system of Lubsey and Gavali to migrate based on priority One having ordinary skill in the art would have been motivated to use Pan into the system of Lubsey and Gavali for the purpose of performing QoS-based scheduling using workload profiles. (Pan paragraph 05)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Pan (US 2021/0019160 A1) and Qi (US 2024/0069974 A1).
As per claim 9, Lubsey and Gavali and Pan do not teach wherein the policy includes a plurality of criteria for rescheduling in the logical cloud and reflection rankings of the plurality of respective criteria.
However, Qi teaches wherein the policy includes a plurality of criteria for rescheduling in the logical cloud and reflection rankings of the plurality of respective criteria. (Qi [0002] Workload scheduling and workload distribution are common functions in the computer field, including in distributed systems. Distributed systems may include, for example, open-source container systems. Open-source container systems such as clusters offer adaptive load balancing, service registration, deployment, operation, resource scheduling, and capacity scaling. [0037] In some embodiments of the present disclosure, the operations may include identifying the first task is within the policy. For example, multiple tasks may be identified, and the first task may be selected from the multiple tasks because the first task satisfies one or more scheduling policy requirements or because the first task ranks the highest in a ranking system established by the scheduling policy requirements).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Qi with the system of Lubsey and Gavali and Pan to reschedule based on rankings. One having ordinary skill in the art would have been motivated to use Qi into the system of Lubsey and Gavali and Pan for the purpose of providing workload management in distributed systems. (Qi paragraph 01)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Pan (US 2021/0019160 A1) and Qi (US 2024/0069974 A1) and Page (US 2006/0165087 A1) and Capano (US 2025/0348399 A1).
As per claim 10, Lubsey and Gavali do not teach wherein the plurality of criteria includes a priority defined in the workload intent.
However, Pan teaches wherein the plurality of criteria includes a priority defined in the workload intent (Pan [0050] This can include migrating the higher-priority workloads prior to the lower-priority workloads. Additionally, at the new host, the priority placement primitive 414 can again be used as part of the initial placement workflow 452, in an example),
Pan does not teach an inter-service connectivity.
However, Page teaches an inter-service connectivity (Page [0067] Through the networking architecture of the present invention, each NPP is connected to one or more service providers, essentially facilitating multi-lateral service provider inter-connectivity. One type of inter-connectivity service involves providing connectivity between peering points. In order to facilitate the connection between two or more peering points, transit gateways are employed. Another type of inter-connectivity service involves providing connectivity between subscriber site(s) and peering point(s). Here, distribution gateways are used. [0071] The interconnect domain further enables: an interconnect that substantially allows a carrier to restrict per VPN the number of routes learned from or advertised to one or more carriers; an interconnect that substantially a carrier to summarize per VPN routes learned from or advertised to one or more carriers; an interconnect that substantially allows a carrier to load balance traffic across one or more networking devices; a single gateway router configuration that allows any IP based VPN carrier to connect to a service grid).
Inter-connectivity is not defined anywhere.
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Page with the system of Lubsey and Gavali and Pan to use inter-service connectivity . One having ordinary skill in the art would have been motivated to use Page into the system of Lubsey and Gavali and Pan for the purpose of providing a networking architecture for configuring a virtual private network. (Page paragraph 44)
Lubsey and Gavali and Pan do not teach a central processing unit (CPU) utilization.
However, Capano teaches a central processing unit (CPU) utilization (Capano [0220] In at least one embodiment, upon detecting a high anomaly likelihood associated with future CPU utilization forecasts, the system triggers a controller to automatically initiate provisioning of additional compute nodes or containers to absorb the projected load, thereby preventing performance degradation. For example, the system calculates future resource usage and anomaly scores, therefore automatically initiating provisioning of additional compute nodes or containers to absorb the projected load leverages that foresight to maintain system stability).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Capano with the system of Lubsey and Gavali and Pan and Page to use central processing unit (CPU) utilization. One having ordinary skill in the art would have been motivated to use Capano into the system of Lubsey and Gavali and Pan and Page for the purpose of iterative method, a device and a system for monitoring a computing device (Capano paragraph 02)
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Cai (US 2025/0097117 A1)
As per claim 11, Lubsey and Gavali do not teach wherein the first cluster and the second cluster are included in different clouds.
However, Cai teaches wherein the first cluster and the second cluster are included in different clouds. (Cai [0012] Multi-cloud platforms may manage multiple clusters across different cloud providers and may provide features like orchestration, load balancing, security, etc., however, the multi-cloud platforms don't have support for global horizontal autoscaling across the managed multiple clusters yet)
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Cai with the system of Lubsey and Gavali to have clusters in different clouds. One having ordinary skill in the art would have been motivated to use Cai into the system of Lubsey and Gavali for the purpose of handling cloud computing in a communication network associated with multiple cloud infrastructure. (Cai paragraph 01)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Lubsey (US 11,226,846 B2) in view of Gavali (US 2020/0364086 A1) in further view of Cai (US 2025/0097117 A1) and Chen (US 2017/0359380 A1)
As per claim 12, Gavali teaches between the migrating of the one or more first workloads to the second cluster and the deleting of the one or more first workloads from the first cluster (Gavali [0052] If the average resource utilization of a worker node is below the lower threshold, then again trigger workload redistribution. Select worker nodes that have higher utilization (i.e., not only over-utilized worker nodes). Select the workloads in these higher utilization worker nodes for migration so that once these selected workloads are added to the under-utilized worker nodes [migration to another cluster], the total resource utilization of these under-utilized worker nodes will raise above the lower threshold. Delete the selected workloads from the higher utilization worker nodes).
Gavali does not teach changing an internet protocol (IP) address mapped with a uniform resource locator (URL) of an initial service to an IP address of a service which is present in the second cluster in a name server; and transmitting traffic input from a first Istio ingress gateway included in the first cluster to a second Istio ingress gateway included in the second cluster.
However, Chen teaches changing an internet protocol (IP) address mapped with a uniform resource locator (URL) of an initial service to an IP address of a service which is present in the second cluster in a name server; and transmitting traffic input from a first Istio ingress gateway included in the first cluster to a second Istio ingress gateway included in the second cluster. (Chen [0078] In another embodiment, once the second or subsequent VPN server is received by the policy controller 415 or the router 420, the traffic of the user equipment is redirected from the first VPN server to the second VPN server. The IP addresses of the first and second servers are different. This dynamically changing IP addresses based upon a predetermined condition, whether time or event based, helps improve the security of network employing VPN servers to access the internet).
It would have been obvious to a person in the ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with the system of Lubsey and Gavali to change an ip address. One having ordinary skill in the art would have been motivated to use Chen into the system of Lubsey and Gavali for the purpose of managing policies to control the frequency of employing new VPN servers. (Chen paragraph 02)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20240427645 A1 – discloses rebalancing in a fleet of storage systems using data science including generating, by the cloud-based rebalancing system, a plurality of workload migration scenarios to address a detected workload imbalance among a plurality of workloads in a fleet of storage systems; determining, by the cloud-based rebalancing system, a plurality of movement vectors for each workload migration scenario, wherein each of the plurality of movement vectors describes a consideration factor for migrating a workload of the plurality of workloads within the fleet of storage systems; and generating, by the cloud-based rebalancing system, at least one rebalancing proposal based on the plurality of movement vectors for each workload migration scenario.
US 20240202019 A1 – discloses r migrating containerized workloads across different container orchestration platform offerings. The method generally includes receiving a migration specification for the workloads identifying at least a source cluster and a destination cluster, wherein the source and destination clusters are provisioned via different container orchestration platform offerings; obtaining a current state of the workloads running on the source cluster based on objects created for the source cluster, wherein the objects comprise a first object supported by the first orchestration platform offering of the source cluster and not the second orchestration platform offering of the destination cluster; applying mutation logic to convert the first object to a second object supported by the second orchestration platform offering of the destination cluster; storing one or more images associated with the containerized workloads on the destination cluster; and configuring the workloads at the destination cluster using the second object.
US 20230208914 A1 – discloses live migration from a first cluster to a second cluster. For instance, when requests to one or more cluster control planes are received, a predetermined fraction of the received requests may be allocated to a control plane of the second cluster, while a remaining fraction of the received requests may be allocated to a control plane of the first cluster. The predetermined fraction of requests are handled using the control plane of the second cluster. While handling the predetermined fraction of requests, it is detected whether there are failures in the second cluster. Based on not detecting failures in the second cluster, the predetermined fraction of requests allocated to the control plane of the second cluster may be increased in predetermined stages until all requests are allocated to the control plane of the second cluster.
US 20220188172 A1 – discloses receiving a request to deploy a workload in a container environment, where: the container environment comprises a plurality of external providers running container environment clusters; and the request (i) includes one or more requirements of the workload and (ii) does not specify a particular external provider of the plurality of external providers. A processor determines a cluster, from the plurality of external providers running the container environment clusters, that meets the one or more requirements of the workload. A processor deploys the workload on the determined cluster.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEHRAN KAMRAN whose telephone number is (571)272-3401. The examiner can normally be reached on 9-5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, April Blair can be reached on (571)270-1014. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEHRAN KAMRAN/ Primary Examiner, Art Unit 2196
Prosecution Insights