Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to Applicant’s amendment filed on 04/01/2026. Claims 2, 14,
and 20 have been cancelled. Claims 1, 13, and 19 have been amended. New claim 21 has been added. Therefore, Claims 1, 3-13, 15-19 and 21 are pending. Any examiner’s note, objection, or rejection not repeated is withdrawn due to Applicant’s amendment.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-13, 15-19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Sivakumar et al. (US 20220279421 A1) in view of Ho (US 20160378545 A1), hereinafter referred to as Sivakumar and Ho, respectively.
Regarding Claim 1, Sivakumar discloses A system, comprising: a device comprising a plurality of cores and memory ([0164] an example computing device (e.g., host), according to techniques described in this disclosure. […] Computing device 800 includes in this example, a bus 242 coupling hardware components of a computing device 800 hardware environment. Bus 242 couples network interface card (NIC) 230, storage disk 246, and one or more microprocessors 210 (hereinafter, “microprocessor 810”). […] A front-side bus may in some cases couple microprocessor 810 and memory device 244. Please note the computing device 800 including coupled microprocessors 210 and memory device 244 corresponds to Applicant’s system comprising a device comprising a plurality of cores and memory.) to:
maintain a cluster of containers in a user space separate from a kernel space of the device, each container in the cluster of containers to execute a respective one of a plurality of virtual functions ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process.; [0071] Containers, including those implementing containerized routing protocol daemons 24, may be deployed to a virtualization environment using a cluster-based framework. Please note that containers deployed to a virtualization environment using a cluster-based framework, where the containers are executing a cRPD in the user space, corresponds to maintaining a cluster of containers in a user space separate from a kernel space of the device, where each container in the cluster executes a respective one of a plurality of virtual functions. Since each container runs a cRPD process that performs respective processing in a virtualized environment, this corresponds to executing respective virtual functions.),
for a network interface card of the device, configured to cause packets received by the device to bypass the kernel space ([0019] The cRPD also programs the data plane on each compute node. For better network packet I/O performance, the DU application may run in the application Pod to bypass the kernel networking stack and abstractions, and thereby use, e.g., zero-copy mechanisms to directly send/receive packets from the physical NIC. Please note that directly receiving packets from the physical NIC by bypassing the kernel networking stack corresponds to Applicant’s causing packets received by the device to bypass the kernel space for the NIC of the device.);
forward, via a load balancing technique, a packet received by the device to a container in the cluster of containers in the user space that executes a virtual function of the plurality of virtual functions ([0086] Orchestrator 50 orchestrates pods comprising container workloads. CNI 312 configures virtual interfaces between pods and the data plane, which may be DPDK-based vRouter 206A. […] In some examples, vRouter 206A has a bonded interface to NIC 321B, which may be an Intel-based NIC that supports DPDK. Bonded interfaces facilitate packet load balancing among fabric interfaces.; [0179] In a DPDK-based deployment of virtual router 206A (shown in FIG. 2), virtual router 206A is installed as a user space 245 application that is linked to the DPDK library. This may lead to faster performance than a kernel-based deployment, particularly in the presence of high packet rates. […] This includes packet polling, packet processing, and packet forwarding. In other words, user packet processing steps are performed by the virtual router 206A DPDK data plane. Please note that the CNI configuring virtual interfaces between pods, comprising container workloads, and the data plane which is DPDK-based vRouter 206A, which has a bonded interface that facilitates packet load balancing, where the virtual router 206A DPDK data plane performs packet processing steps, corresponds to Applicant’s forwarding a packet received by the device to a container in the cluster of containers in the user space that executes a virtual function of the plurality of virtual functions via a load balancing technique. This is because the virtual router 206A DPDK data plane corresponds to the load balancing technique, and forwards the packets received by the device to the respective containers in the pod corresponding to cluster of containers via the virtual interface.);
Sivakumar does not explicitly disclose identify a number of cores accessible to the device to process packets received by the device;
determine, responsive to the number of cores greater than a threshold, to establish a predetermined number of containers for the plurality of virtual functions;
and configure, based on the number of cores, each of the plurality of virtual functions with a predetermined number of queues, wherein each queue of the predetermined number of queues maps to a core of the plurality of cores;
and update a queue for a core of the plurality of cores managed by the virtual function to cause the core to process the packet in accordance with the queue.
However, Ho discloses identify a number of cores accessible to the device to process packets received by the device ([0300] applications of group 22, container 91 an/or multithreaded application 93 may be processing workloads generated from clients or server applications—using the OS managed processer and hardware resources (e.g., CPU/core cycles, memories, and network and I/O ports/interfaces)—to produce useful results. For each “unit of workload” (henceforth, shortened to “workload”), an application needs to process to produce results, and as incoming workloads get assigned to an application on an ongoing basis, this processing can be modeled and may be implemented as a queue of workloads in a software processing queue, such as workload processing queues 107 illustrated in SMP OS kernel 46. In workload processing queues 107, first in, first out (FIFO) queues, such as […] packet queues 73 […] may be continually being emptied by the application (such as applications of group 22, container 91 and/or 93). Please note that the core cycles of the OS managed processor that is available for workload processing queues 107 such as packet queues 73 corresponds to identifying a number of cores accessible to the device to process packets received by the device.);
determine, responsive to the number of cores greater than a threshold, to establish a predetermined number of containers for the plurality of virtual functions ([0292] for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […] These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Please note that an aggregated threshold with multiple queue thresholds for multiple software processing queues based on CPU core utilization corresponds to determining to establish a predetermined number of containers for the plurality of virtual functions responsive to the number of cores greater than a threshold, as each container processes a respective workload that has a software processing queue, and therefore the available core cycles being higher, in effect being greater than the queue threshold, would result in a proportional number of containers being established to operate using those core cycles.);
and configure, based on the number of cores, each of the plurality of virtual functions with a predetermined number of queues, wherein each queue of the predetermined number of queues maps to a core of the plurality of cores ([0290] In this model, each application's thread of execution is continually processing workloads (per their abstractions, representations, and data in the queues) from parallel queues to produce results, operating within the constraints of the resources (e.g., CPU/cores, memory, and storage, etc.) assigned to it either dynamically or statically.
Please note that the thread of execution of each application operating parallel queues within the constraints of the cores dynamically assigned to it corresponds to configuring each of the plurality of virtual functions with a predetermined number of queues where each queue of the predetermined number of queues maps to a core of the plurality of cores based on the number of cores. This is because the number of queues of each application is configured based on the number of cores, i.e., the resource constraints, and each application corresponds to each of the plurality of virtual functions and is mapped to the core that is executing it.).
and update a queue for a core of the plurality of cores managed by the virtual function to cause the core to process the packet in accordance with the queue ([0396] Referring now to FIG. 17, as illustrated in workload tuning operation 163, scheduler 114 may cause PRT 25 to schedule or reschedule data transfers with various different software processing queues in queues 82 in accordance with dynamic workload changes, e.g. during processing of application 93 by core 99 […] data between the software queues and its relevant external entities (e.g., hardware queues of input/output packets in network interface cards). Please note that software processing queues operating with hardware queues of input/output packets being scheduled by scheduler 114 during processing of application 93 by core 99 corresponds to updating a queue to cause processing of the packet in accordance with the queue, and since the queue is run on a particular core of the cores managed by the system, this corresponds to the queue being for the core and the core processing the packet.).
Sivakumar and Ho are both considered to be analogous to the claimed invention because they are in the same field of containerized computer packet processing. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Sivakumar to incorporate the teachings of Ho to modify the system maintaining container clusters in a user space separate from the kernel space, each container executing respective virtual functions, and forwarding packets received by the device to containers via a load balancing technique to identify a number of cores accessible to the device to process packets received by the device, determine to establish a predetermined number of containers for the virtual functions if the number of cores is greater than a threshold, configure each of the virtual functions with a predetermined number of queues based on the number of cores, wherein each queue of the predetermined number of queues maps to a core of the plurality of cores, and update a queue for a core of the cores managed by the virtual function to cause it to process the packet, allowing for improved packet processing performance and resource usage efficiency, as described in Ho.
Regarding Claim 3, Sivakumar-Ho as described in Claim 1, Ho further discloses identify a configuration file established for the device, the configuration file comprising an indication of a number of cores of the device, an auto-scale factor, a scale-up threshold, and a scale down threshold ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […]These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Queue threshold can also be configured manually, instead of automatically via statistical analysis of measured data, etc.
Please note that the configured queue threshold data corresponds to the identified configuration file established for the device, as it is known in the art that data defining parameters of operation of a device can be stored as a file. In order to determine the CPU core utilization, it is inherently necessary to have an indication of a number of cores of the device. Furthermore, the processing queue threshold that is determined automatically via statistical analysis of measured data corresponds to the auto-scale factor. Additionally, the multiple queue thresholds for multiple software processing queues correspond to the scale-up and scale down thresholds, as it is obvious to one of ordinary skill in the art to set thresholds that either scale down or up the system in order to maximize CPU core utilization in a balanced manner across the system.).
Regarding Claim 4, Sivakumar-Ho as described in Claim 1, Sivakumar further discloses a daemon executed by the device in the user space, the daemon configured to: split the network interface card of the device into the plurality of virtual functions ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process; [0086] vRouter 206A has a bonded interface to NIC 321B, which may be an Intel-based NIC that supports DPDK. Bonded interfaces facilitate packet load balancing among fabric interfaces.; [0112] a virtual router agent 314 for the virtual router data plane may operate as a gRPC server 520 that exposes gRPC APIs for programming the virtual router data plane 206A. The techniques include workflows for configuring virtual network interfaces for pods, where the virtual router agent 314 obtains the information from a containerized routing protocol daemon (cRPD) 324. Please note that the cRPD executed in the user space corresponds to the daemon executed by the device in the user space, and since the virtual router agent 314 configures virtual network interfaces for pods based on information obtained from the cRPD, and uses an API for programming the virtual router data plane 206A which has a bonded interface to NIC 321B, this corresponds to splitting the NIC into the plurality of virtual functions.);
and configure a plurality of containers of the cluster of containers with a respective one of the plurality of virtual functions to cause packets received by the device to bypass the kernel space and be processed by at least one of the plurality of virtual functions configured in the plurality of containers ([0070] the data plane of vCSRs 20 is also containerized and orchestrated by orchestrator 50. The data plane may be a DPDK-based virtual router, for instance.; [0071] Containers, including those implementing containerized routing protocol daemons 24, may be deployed to a virtualization environment using a cluster-based framework; [0291] DPDK enables building applications that can bypass the kernel for packet I/O.
Please note that the containers being deployed using a cluster-based framework, where the data plane is a DPDK-based virtual router that is containerized, and the DPDK allowing bypassing the kernel for packet I/O, corresponds to configuring a plurality of containers of the cluster of containers with a respective one of the plurality of virtual functions to cause packets received by the device to bypass the kernel space and be processed by at least one of the plurality of virtual functions configured in the plurality of containers. This is because the applications in the virtualization environment in each container, corresponding to virtual functions configured in containers, implement the cRPD, and therefore the configuration is caused by the daemon.).
Regarding Claim 5, Sivakumar-Ho as described in Claim 4, Ho further discloses identify, in a configuration file established for the device, a number of cores accessible to the device and an indication to auto-scale ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […]These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Queue threshold can also be configured manually, instead of automatically via statistical analysis of measured data, etc. Please note that, as previously stated, the configured queue threshold data corresponds to the identified configuration file established for the device, as it is known in the art that data defining parameters of operation of a device can be stored as a file, in order to determine the CPU core utilization, it is inherently necessary to have an indication of a number of cores of the device, and the processing queue threshold that is determined automatically via statistical analysis of measured data corresponds to the indication to auto-scale. );
and invoke, responsive to the indication to auto-scale and the number of cores greater than a threshold ([0333] Alternately, statistical computed queue threshold 148 may involve application-specific measurement and analysis either online or off-line, in which an instance of the application, such as application, process or thread 85, may be executed that fully utilizes all resources of a normalized resource set (e.g., of CPU/core cycles, memories, networking, etc.) […] Queue threshold 148 can be computed as a function of the resulting measured/tested average. Please note that the queue threshold 148 being computed as a function of the resulting measured/tested average, where the application is fully utilizing core cycles, corresponds to invoking responsive to the indication to auto-scale and the number of cores greater than a threshold, as the operation is based on the threshold and the available resources are based on the auto-scaling resources.),
Sivakumar further discloses the daemon ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process. Thus, cRPD makes available the rich routing software pedigree of physical routers on Linux-based compute nodes. cRPD provides control plane functionality. Please note that since the containers have the cRPD packaged within them, this corresponds to the daemon being invoked, as it is run with the containers as the scaling occurs.).
Regarding Claim 6, Sivakumar-Ho as described in Claim 1, Ho further discloses invoke, responsive to one or more parameters indicated in a configuration file established for the device, a daemon configured to determine a number of virtual functions operable with the network interface card of the device, a number of threads operable by each virtual function, and a maximum number of containers operable within the cluster of containers ([0009] SMP OS is loaded onto a computer system as the host OS, the OS is typically loaded into main memory; [0146] These micro-virtualization engines, through their micro-virtualization execution framework, can process selective or all software calls, events, and call backs for the container(s) specific to a processor core. In this way, execution, data, and event parallelization and parallelism are maximized over containers running over many cores; [0298] Multiple, concurrent, and strongly application-associative software processing queues may each be mapped and bounded to each of an application's threads of execution (processes or threads or other execution abstractions), for one or more applications running concurrently on the SMP OS, which in turn may run with or without a hypervisor over one or more shared memory multi-core processors. […] The result is that in situ performance profiles, workload handling, and QoE/QoS of the applications and their individual threads of execution can be measured and analyzed individually (and obviously also in totality) on an SMP OS for granular monitoring and resource management in real time and in situ.; [0378] The operation of scheduling decisions 151 for latency-sensitive applications, applied by dynamic resource scheduler 114 […] to the host OS; [0380] Resource scheduler 114 enforces decisions 151 by relying on hardware control features (e.g., rate limiting and filtering capability of one or more of the NICs).
Please note that the SMP OS loaded onto the computer system via memory corresponds to the daemon invoked responsive to parameters indicated in a configuration file established for the device, as the necessary parameters for its operation would be stored in configuration files of the memory. Additionally, the system maximizing parallelism for containers running over many cores corresponds to determining a maximum number of containers operable within the cluster of containers as it would necessitate the determination of the maximum number of containers that could run over the cores in order to be maximally parallel. Furthermore, since the workload handling and individual threads of execution of the applications can be analyzed individually corresponds to determining a number of virtual functions and a number of threads operable by each virtual function. Lastly, since the resource scheduler 114 associated with the decisions 151 applied to the OS controls the NIC, this corresponds to the virtual functions being operable with the NIC of the device.);
determine a number of cores the daemon is capable to support based on the number of virtual functions, the number of threads, and the maximum number of containers ([0101] Additional processing elements 25, such as emulated kernel services 44, kernel-space parallel processing 52 and user-space buffers 48, may be loaded into user-space 17 and/or kernel space 19 of main memory 18, and/or otherwise made available for processing in one or more of the cores of at least one multi-core processor, such as core 0 multi-core processor 12, to substantially improve processing performance and processing time of software applications, software application groups, and containers running concurrently and/or sequentially under control of the SMP OS and its cores; [0339] Resource scheduler 114, may be in the same or different context with application 87 and PRT 25, decides the distribution of resource to be made available to application 87 and/or PRT 25 and/or other modules, such as buffers 48, in application group 24. Please note that the resource scheduler 114 deciding the distribution of resources to be made available to applications 87 in order to improve processing performance of containers running concurrently under control of the SMP OS and its cores corresponds to Applicant’s determining a number of cores the daemon, i.e., the SMP OS, is capable to support based on the number of virtual functions, number of threads, and maximum number of containers, as these values are used by the resource scheduler 114 in consideration of resource distribution to be made available to applications, in this case, determining the number of cores the SMP OS can support for the containers.);
and establish, based on a comparison of the determined number of cores the daemon is capable to support with a number of cores accessible by the device, the cluster of containers ([0369] PRT 25, scheduler 114 and application 93 all run on core 98 and therefore must share the available CPU cycles, e.g. of core 98. Thus, it is desirable to achieve a balance between the resource consumption of scheduler 114, PRT 25 and application 93 to maximize the performance of application 93. The use of groups of programs, related by their types of resource consumption such as groups or containers 22, 24 and 26, and PRT 25 substantially reduces the resource consumption of application 93. Please note that balancing resource consumption to maximize the performance of application 93 in containers by sharing available CPU cycles of core 98 corresponds to establishing the cluster of containers based on a comparison of the determined number of cores the daemon is capable to support with a number of cores accessible by the device, as even if the application 93 running in the established cluster of containers 22, 24, and 26 could be supported further by the capability of the scheduler 114, this is compared to the number of cores accessible to the device, i.e., the available CPU cycles of core 98, which in turn affects the establishment of the containers to balance resource consumption.).
Regarding Claim 7, Sivakumar-Ho as described in Claim 1, Sivakumar further discloses containers within the cluster of containers communicate via an internal bridge network that is not directly accessible via a public network, and the device is further configured to assign an internet protocol address to the cluster of containers ([0170] Network interface card 230 may also implement SR-IOV to enable sharing the physical network function (I/O) among one or more virtual execution elements, such as containers 229A-229B or one or more virtual machines (not shown in FIG. 2). Shared virtual devices such as virtual functions may provide dedicated resources such that each of the virtual execution elements may access dedicated resources of NIC 230, which therefore appears to each of the virtual execution elements as a dedicated NIC. Virtual functions may represent lightweight PCIe functions that share physical resources with a physical function used by physical driver 225 and with other virtual functions.; [0180] In general, each of pods 202A-202B may be assigned one or more virtual network addresses for use within respective virtual networks, where each of the virtual networks may be associated with a different virtual subnet provided by virtual router 206A.
Please note that the virtual functions of the containers representing PCIe functions that share resources with other virtual functions corresponds to containers within the cluster of containers communicating via an internal bridge network that is not directly accessible via a public network, as Applicant states in [00102] of the Specification that “the internal bridge network 104 can be a Peripheral Component Interconnect Express (PCIe) network 104,” which is what the virtual functions of containers, not publicly accessible, are using. Furthermore, each pod being assigned virtual network addresses for use within respective virtual networks, each associated with a different virtual subnet corresponds to assigning an internet protocol address to the cluster of containers.).
Regarding Claim 8, Sivakumar-Ho as described in Claim 1, Ho further discloses identify a utilization of a first plurality of cores managed by the plurality of virtual functions ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration. Please note that since the CPU core utilization is measured while the application process runs over a set duration, this corresponds to identifying a utilization of a first plurality of cores managed by the plurality of virtual functions.);
determine, responsive to the utilization greater than or equal to a threshold, to upscale the cluster of containers ([0092] processing may be scheduled across multiple CPU cores to achieve higher core and overall processor utilization; [0373] PRT 25, and/or software services 47, may actively alter the resource allocation of core 98 to increase or decrease the number or percentage of CPU cycles to be provided for execution of application 93 […] Resource scheduler 114 may allocate new or additional resources, such as additional CPU cycles of core 98, for processing application 93 if scheduler 114 determines or predicts resource bottlenecks that may, for example, interfere with achievement of QoS requirements 206 of application 93 which cannot otherwise be resolved by resource scheduler 114 using resources then currently in use.
Please note that altering the resource allocation to increase the number of CPU cycles to be provided for execution of application 93 in order to achieve higher core utilization corresponds to determining to upscale the cluster of containers responsive to the utilization greater than or equal to a threshold, as if the cycles of a core provided to an application were not sufficient and causing a bottleneck, i.e., greater than or equal to a threshold, the amount of dedicated resources of the cluster of containers would be upscaled.);
Sivakumar further discloses invoke, responsive to the determination to upscale the cluster of containers, an additional container with an additional virtual function for an additional core of the plurality of cores ([0162] vRouter agent 314 listens for Port-Add and Port-Delete messages, e.g., on a thrift service, where a “port” corresponds to a virtual network interface. CNI 312 sends a Port-Add message to vRouter agent 314 (702). The Port-Add message includes an identifier for the virtual network for the port and an IP address allocated by CNI 312 for the Pod. Please note that Port-Add message being sent to the vRouter agent 314, where an IP address is allocated for the Pod and an identifier for the virtual network for the port is included, corresponds to invoking an additional container with an additional function for an additional core of the plurality of cores responsive to the determination to upscale the cluster of containers, as it invokes the virtual interfaces and establishes the IP address needed for the additional container of the cluster of containers.);
and add the additional container to the cluster of containers ([0081] a CNI of server 12A, when triggered by pod events from orchestrator 50, dynamically adds or deletes virtual network interfaces between the pod (here deployed with DU 22A) and the vRouter 206A, which may also be deployed as container in some examples; [0149] A Port Add and Port Down sequence may be primarily done via cRPD or via a vRouter Agent. Example such sequences for Port Add and Port Down are below: […] Port Add 1. CNI has the IP address block reserved for Pods 2. On port-add allocates IP and configures container. Please note that the CNI allocating the IP and configuring the container as part of the port-add to dynamically add virtual interfaces between the pod and the vRouter 206A, deployed as a container, corresponds to adding the additional container to the cluster of containers, i.e., pod.).
Regarding Claim 9, Sivakumar-Ho as described in Claim 1, Ho further discloses identify a utilization of the plurality of cores managed by the plurality of virtual functions ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration.
Please note that since the CPU core utilization is measured while the application process runs over a set duration, this corresponds to identifying a utilization of a first plurality of cores managed by the plurality of virtual functions.);
determine, responsive to the utilization less than or equal to a threshold, to downscale the cluster of containers ([0092] processing may be scheduled across multiple CPU cores to achieve higher core and overall processor utilization; [0373] PRT 25, and/or software services 47, may actively alter the resource allocation of core 98 to increase or decrease the number or percentage of CPU cycles to be provided for execution of application 93 […] Resource scheduler 114 may allocate new or additional resources, such as additional CPU cycles of core 98, for processing application 93 if scheduler 114 determines or predicts resource bottlenecks that may, for example, interfere with achievement of QoS requirements 206 of application 93 which cannot otherwise be resolved by resource scheduler 114 using resources then currently in use. Please note that altering the resource allocation to decrease the number of CPU cycles to be provided for execution of application 93 in order to achieve higher core utilization corresponds to determining to downscale the cluster of containers responsive to the utilization less than or equal to a threshold, as if the cycles of a core provided to an application were not fully being utilized, i.e., less than or equal to a threshold, the amount of dedicated resources of the cluster of containers would be downscaled.);
Sivakumar further discloses and cause removal of at least one container from the cluster of containers to reduce a number of containers in the cluster of containers ([0081] a CNI of server 12A, when triggered by pod events from orchestrator 50, dynamically adds or deletes virtual network interfaces between the pod (here deployed with DU 22A) and the vRouter 206A, which may also be deployed as container in some examples; [0149] A Port Add and Port Down sequence may be primarily done via cRPD or via a vRouter Agent. Example such sequences for Port Add and Port Down are below: […] Port Down 1. DPDK informs Agent 2. Agent sends VMI-Down to cRPD 3. cRPD withdraws route/NH. Please note that the CNI dynamically deleting the virtual network interface between the pod and the vRouter 206A, deployed as a container, by carrying out the Port Down and withdrawing the route corresponds to causing the removal of at least one container from the cluster of containers, i.e., pod, to reduce a number of containers in the cluster of containers.).
Regarding Claim 10, Sivakumar-Ho as described in Claim 1, Sivakumar further discloses detect an error associated with the container or the virtual function in the cluster of containers ([0077] Kubernetes permits leveraging its existing mechanisms for monitoring the health of containerized RPD 24s and restarting if necessary.
Please note that monitoring the health of containerized RPD 24s corresponds to detecting an error associated with the container in the cluster of containers, as it is known in the art that health monitoring mechanisms detect errors.);
and replace, responsive to detection of the error, the container with a new container and a new virtual function to manage the core ([0077] Kubernetes permits leveraging its existing mechanisms for monitoring the health of containerized RPD 24s and restarting if necessary. Please note that restarting the containerized RPD 24s as needed based on health monitoring corresponds to replacing the container with a new container and a new virtual function to manage the core responsive to the detection of the error, as it would re-initialize the container with its associated virtual function as part of restarting it when needed based on health monitoring, i.e., responsive to detecting an error.).
Regarding Claim 11, Sivakumar-Ho as described in Claim 1, Sivakumar further discloses detect an error associated with the container or the virtual function in the cluster of containers ([0114] the vRouter (or dpdk) forwarding plane. The vRouter agent may be a user space process running on Linux. It acts as the local, lightweight control plane and is responsible for the following functions: [0115] It exchanges control states such as routes with the control nodes using XMPP. [0116] It receives low-level configuration state such as routing instances and forwarding policy from the control nodes using XMPP. [0117] It reports analytics state such as logs, statistics, and events to the analytics nodes. Please note that analytics states such as events being gathered as part of the function of the vRouter agent control plane corresponds to detecting an error associated with the container in the cluster of containers, as it is known in the art that an error is a computer event.);
include the error in a log for the container ([0114] the vRouter (or dpdk) forwarding plane. The vRouter agent may be a user space process running on Linux. It acts as the local, lightweight control plane and is responsible for the following functions: [0115] It exchanges control states such as routes with the control nodes using XMPP. [0116] It receives low-level configuration state such as routing instances and forwarding policy from the control nodes using XMPP. [0117] It reports analytics state such as logs, statistics, and events to the analytics nodes. Please note that analytics states such as events and logs being gathered as part of the function of the vRouter agent control plane reporting them corresponds to including the error in a log, as it is known in the art that an error is a computer event, and could be included in the gathered logs to be reported for analysis.);
and provide the log to a technical support device remote from the device ([0117] It reports analytics state such as logs, statistics, and events to the analytics nodes. Please note that reporting the analytics state to the analytics nodes including the logs and events corresponds to providing the log to a technical support device remote from the device, as the analytics nodes correspond to the technical support device remote from the device, i.e., control plane.).
Regarding Claim 12, Sivakumar-Ho as described in Claim 1, Sivakumar further discloses store, in a log, an indication of an error associated with the container or the virtual function in the cluster of containers ([0114] the vRouter (or dpdk) forwarding plane. The vRouter agent may be a user space process running on Linux. It acts as the local, lightweight control plane and is responsible for the following functions: [0115] It exchanges control states such as routes with the control nodes using XMPP. [0116] It receives low-level configuration state such as routing instances and forwarding policy from the control nodes using XMPP. [0117] It reports analytics state such as logs, statistics, and events to the analytics nodes.
Please note that analytics states such as events and logs being gathered as part of the function of the vRouter agent control plane corresponds to storing an error associated with the container in the cluster of containers in a log, as it is known in the art that an error is a computer event, and could be included in the gathered logs to be reported for analysis.);
provide the log to a technical support device remote from the device ([0117] It reports analytics state such as logs, statistics, and events to the analytics nodes. Please note that reporting the analytics state to the analytics nodes including the logs and events corresponds to providing the log to a technical support device remote from the device, as the analytics nodes correspond to the technical support device remote from the device, i.e., control plane.);
and cause, responsive to the error and subsequent to provision of the log to the technical support device, removal of the container from the cluster of containers ([0020] a DPDK-based virtual router to support DPDK applications. A CNI plugin manages the DPDK configuration for application and programs the virtual router. This may include setting up a vhost control channel and assigning IP (e.g., both IPv4 and IPv6) and MAC addresses, advertising the Pod IP addresses, and detecting and withdrawing the routes when the Pod is considered down or removed. Please note that withdrawing the routes when the Pod is considered down or removed, as part of the operation of the virtual router, corresponds to removing the container from the cluster of containers responsive to the error and subsequent to provision of the log to the technical support device, as the vRouter agent reports events such as errors, which may indicate the pod is down, and therefore cause the removal of the container from the cluster of containers responsive to the error.).
Regarding Claim 13, Sivakumar discloses A method, comprising: maintaining, by a device comprising a plurality of cores and memory, a cluster of containers in a user space separate from a kernel space of the device, each container in the cluster of containers to execute a respective one of a plurality of virtual functions ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process.; [0071] Containers, including those implementing containerized routing protocol daemons 24, may be deployed to a virtualization environment using a cluster-based framework. Please note that containers deployed to a virtualization environment using a cluster-based framework, where the containers are executing a cRPD in the user space, corresponds to maintaining a cluster of containers in a user space separate from a kernel space of the device, where each container in the cluster executes a respective one of a plurality of virtual functions. Since each container runs a cRPD process that performs respective processing in a virtualized environment, this corresponds to executing respective virtual functions.),
for a network interface card of the device, configured to cause packets received by the device to bypass the kernel space ([0019] The cRPD also programs the data plane on each compute node. For better network packet I/O performance, the DU application may run in the application Pod to bypass the kernel networking stack and abstractions, and thereby use, e.g., zero-copy mechanisms to directly send/receive packets from the physical NIC. Please note that directly receiving packets from the physical NIC by bypassing the kernel networking stack corresponds to Applicant’s causing packets received by the device to bypass the kernel space for the NIC of the device.);
forwarding, by the device via a load balancing technique, a packet received by the device to a container in the cluster of containers in the user space that executes a virtual function of the plurality of virtual functions ([0086] Orchestrator 50 orchestrates pods comprising container workloads. CNI 312 configures virtual interfaces between pods and the data plane, which may be DPDK-based vRouter 206A. […] In some examples, vRouter 206A has a bonded interface to NIC 321B, which may be an Intel-based NIC that supports DPDK. Bonded interfaces facilitate packet load balancing among fabric interfaces.; [0179] In a DPDK-based deployment of virtual router 206A (shown in FIG. 2), virtual router 206A is installed as a user space 245 application that is linked to the DPDK library. This may lead to faster performance than a kernel-based deployment, particularly in the presence of high packet rates. […] This includes packet polling, packet processing, and packet forwarding. In other words, user packet processing steps are performed by the virtual router 206A DPDK data plane. Please note that the CNI configuring virtual interfaces between pods, comprising container workloads, and the data plane which is DPDK-based vRouter 206A, which has a bonded interface that facilitates packet load balancing, where the virtual router 206A DPDK data plane performs packet processing steps, corresponds to Applicant’s forwarding a packet received by the device to a container in the cluster of containers in the user space that executes a virtual function of the plurality of virtual functions via a load balancing technique. This is because the virtual router 206A DPDK data plane corresponds to the load balancing technique, and forwards the packets received by the device to the respective containers in the pod corresponding to cluster of containers via the virtual interface.);
Sivakumar does not explicitly disclose identifying, by the device, a number of cores accessible to the device to process packets received by the device;
determining, by the device, responsive to the number of cores greater than a threshold, to establish a predetermined number of containers for the plurality of virtual functions;
and configuring, by the device based on the number of cores, each of the plurality of virtual functions with a predetermined number of queues, where in each queue of the predetermined number of queues maps to a core of the plurality of cores;
and updating, by the device, a queue for a core of the plurality of cores managed by the virtual function to cause the core to process the packet in accordance with the queue.
However, Ho discloses identifying, by the device, a number of cores accessible to the device to process packets received by the device ([0300] applications of group 22, container 91 an/or multithreaded application 93 may be processing workloads generated from clients or server applications—using the OS managed processer and hardware resources (e.g., CPU/core cycles, memories, and network and I/O ports/interfaces)—to produce useful results. For each “unit of workload” (henceforth, shortened to “workload”), an application needs to process to produce results, and as incoming workloads get assigned to an application on an ongoing basis, this processing can be modeled and may be implemented as a queue of workloads in a software processing queue, such as workload processing queues 107 illustrated in SMP OS kernel 46. In workload processing queues 107, first in, first out (FIFO) queues, such as […] packet queues 73 […] may be continually being emptied by the application (such as applications of group 22, container 91 and/or 93). Please note that the core cycles of the OS managed processor that is available for workload processing queues 107 such as packet queues 73 corresponds to identifying a number of cores accessible to the device to process packets received by the device.);
determining, by the device, responsive to the number of cores greater than a threshold, to establish a predetermined number of containers for the plurality of virtual functions ([0292] for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […] These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Please note that an aggregated threshold with multiple queue thresholds for multiple software processing queues based on CPU core utilization corresponds to determining to establish a predetermined number of containers for the plurality of virtual functions responsive to the number of cores greater than a threshold, as each container processes a respective workload that has a software processing queue, and therefore the available core cycles being higher, in effect being greater than the queue threshold, would result in a proportional number of containers being established to operate using those core cycles.);
and configuring, by the device based on the number of cores, each of the plurality of virtual functions with a predetermined number of queues, where in each queue of the predetermined number of queues maps to a core of the plurality of cores ([0290] In this model, each application's thread of execution is continually processing workloads (per their abstractions, representations, and data in the queues) from parallel queues to produce results, operating within the constraints of the resources (e.g., CPU/cores, memory, and storage, etc.) assigned to it either dynamically or statically. Please note that the thread of execution of each application operating parallel queues within the constraints of the cores dynamically assigned to it corresponds to configuring each of the plurality of virtual functions with a predetermined number of queues where each queue of the predetermined number of queues maps to a core of the plurality of cores based on the number of cores. This is because the number of queues of each application is configured based on the number of cores, i.e., the resource constraints, and each application corresponds to each of the plurality of virtual functions and is mapped to the core that is executing it.).
and updating, by the device, a queue for a core of the plurality of cores managed by the virtual function to cause the core to process the packet in accordance with the queue ([0396] Referring now to FIG. 17, as illustrated in workload tuning operation 163, scheduler 114 may cause PRT 25 to schedule or reschedule data transfers with various different software processing queues in queues 82 in accordance with dynamic workload changes, e.g. during processing of application 93 by core 99 […] data between the software queues and its relevant external entities (e.g., hardware queues of input/output packets in network interface cards). Please note that software processing queues operating with hardware queues of input/output packets being scheduled by scheduler 114 during processing of application 93 by core 99 corresponds to updating a queue to cause processing of the packet in accordance with the queue, and since the queue is run on a particular core of the cores managed by the system, this corresponds to the queue being for the core and the core processing the packet.).
Sivakumar and Ho are both considered to be analogous to the claimed invention because they are in the same field of containerized computer packet processing. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Sivakumar to incorporate the teachings of Ho to modify the system maintaining container clusters in a user space separate from the kernel space, each container executing respective virtual functions, and forwarding packets received by the device to containers via a load balancing technique to identify a number of cores accessible to the device to process packets received by the device, determine to establish a predetermined number of containers for the virtual functions if the number of cores is greater than a threshold, configure each of the virtual functions with a predetermined number of queues based on the number of cores, wherein each queue of the predetermined number of queues maps to a core of the plurality of cores, and update a queue for a core of the cores managed by the virtual function to cause it to process the packet, allowing for improved packet processing performance and resource usage efficiency, as described in Ho.
Regarding Claim 15, Sivakumar-Ho as described in Claim 13, Ho further discloses identifying, by the device, a configuration file established for the device, the configuration file comprising an indication of a number of cores of the device, an auto-scale factor, a scale-up threshold, and a scale down threshold ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […]These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Queue threshold can also be configured manually, instead of automatically via statistical analysis of measured data, etc. Please note that the configured queue threshold data corresponds to the identified configuration file established for the device, as it is known in the art that data defining parameters of operation of a device can be stored as a file. In order to determine the CPU core utilization, it is inherently necessary to have an indication of a number of cores of the device. Furthermore, the processing queue threshold that is determined automatically via statistical analysis of measured data corresponds to the auto-scale factor. Additionally, the multiple queue thresholds for multiple software processing queues correspond to the scale-up and scale down thresholds, as it is obvious to one of ordinary skill in the art to set thresholds that either scale down or up the system in order to maximize CPU core utilization in a balanced manner across the system.).
Regarding Claim 16, Sivakumar-Ho as described in Claim 13, Sivakumar further discloses splitting, by a daemon executed by the device in the user space, the network interface card of the device into the plurality of virtual functions ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process; [0086] vRouter 206A has a bonded interface to NIC 321B, which may be an Intel-based NIC that supports DPDK. Bonded interfaces facilitate packet load balancing among fabric interfaces.; [0112] a virtual router agent 314 for the virtual router data plane may operate as a gRPC server 520 that exposes gRPC APIs for programming the virtual router data plane 206A. The techniques include workflows for configuring virtual network interfaces for pods, where the virtual router agent 314 obtains the information from a containerized routing protocol daemon (cRPD) 324. Please note that the cRPD executed in the user space corresponds to the daemon executed by the device in the user space, and since the virtual router agent 314 configures virtual network interfaces for pods based on information obtained from the cRPD, and uses an API for programming the virtual router data plane 206A which has a bonded interface to NIC 321B, this corresponds to splitting the NIC into the plurality of virtual functions.);
and configuring, by the daemon, a plurality of containers of the cluster of containers with a respective one of the plurality of virtual functions to cause packets received by the device to bypass the kernel space and be processed by at least one of the plurality of virtual functions configured in the plurality of containers ([0070] the data plane of vCSRs 20 is also containerized and orchestrated by orchestrator 50. The data plane may be a DPDK-based virtual router, for instance.; [0071] Containers, including those implementing containerized routing protocol daemons 24, may be deployed to a virtualization environment using a cluster-based framework; [0291] DPDK enables building applications that can bypass the kernel for packet I/O. Please note that the containers being deployed using a cluster-based framework, where the data plane is a DPDK-based virtual router that is containerized, and the DPDK allowing bypassing the kernel for packet I/O, corresponds to configuring a plurality of containers of the cluster of containers with a respective one of the plurality of virtual functions to cause packets received by the device to bypass the kernel space and be processed by at least one of the plurality of virtual functions configured in the plurality of containers. This is because the applications in the virtualization environment in each container, corresponding to virtual functions configured in containers, implement the cRPD, and therefore the configuration is caused by the daemon.).
Regarding Claim 17, Sivakumar-Ho as described in Claim 16, Ho further discloses identifying, in a configuration file established for the device, a number of cores accessible to the device and an indication to auto-scale ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […]These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Queue threshold can also be configured manually, instead of automatically via statistical analysis of measured data, etc. Please note that, as previously stated, the configured queue threshold data corresponds to the identified configuration file established for the device, as it is known in the art that data defining parameters of operation of a device can be stored as a file, in order to determine the CPU core utilization, it is inherently necessary to have an indication of a number of cores of the device, and the processing queue threshold that is determined automatically via statistical analysis of measured data corresponds to the indication to auto-scale. );
and invoking, responsive to the indication to auto-scale and the number of cores greater than a threshold ([0333] Alternately, statistical computed queue threshold 148 may involve application-specific measurement and analysis either online or off-line, in which an instance of the application, such as application, process or thread 85, may be executed that fully utilizes all resources of a normalized resource set (e.g., of CPU/core cycles, memories, networking, etc.) […] Queue threshold 148 can be computed as a function of the resulting measured/tested average. Please note that the queue threshold 148 being computed as a function of the resulting measured/tested average, where the application is fully utilizing core cycles, corresponds to invoking responsive to the indication to auto-scale and the number of cores greater than a threshold, as the operation is based on the threshold and the available resources are based on the auto-scaling resources.),
Sivakumar further discloses the daemon ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process. Thus, cRPD makes available the rich routing software pedigree of physical routers on Linux-based compute nodes. cRPD provides control plane functionality. Please note that since the containers have the cRPD packaged within them, this corresponds to the daemon being invoked, as it is run with the containers as the scaling occurs.).
Regarding Claim 18, Sivakumar-Ho as described in Claim 13, Ho further discloses invoking, by the device responsive to one or more parameters indicated in a configuration file established for the device, a daemon configured to determine a number of virtual functions operable with the network interface card of the device, a number of threads operable by each virtual function, and a maximum number of containers operable within the cluster of containers ([0009] SMP OS is loaded onto a computer system as the host OS, the OS is typically loaded into main memory; [0146] These micro-virtualization engines, through their micro-virtualization execution framework, can process selective or all software calls, events, and call backs for the container(s) specific to a processor core. In this way, execution, data, and event parallelization and parallelism are maximized over containers running over many cores; [0298] Multiple, concurrent, and strongly application-associative software processing queues may each be mapped and bounded to each of an application's threads of execution (processes or threads or other execution abstractions), for one or more applications running concurrently on the SMP OS, which in turn may run with or without a hypervisor over one or more shared memory multi-core processors. […] The result is that in situ performance profiles, workload handling, and QoE/QoS of the applications and their individual threads of execution can be measured and analyzed individually (and obviously also in totality) on an SMP OS for granular monitoring and resource management in real time and in situ.; [0378] The operation of scheduling decisions 151 for latency-sensitive applications, applied by dynamic resource scheduler 114 […] to the host OS; [0380] Resource scheduler 114 enforces decisions 151 by relying on hardware control features (e.g., rate limiting and filtering capability of one or more of the NICs). Please note that the SMP OS loaded onto the computer system via memory corresponds to the daemon invoked responsive to parameters indicated in a configuration file established for the device, as the necessary parameters for its operation would be stored in configuration files of the memory. Additionally, the system maximizing parallelism for containers running over many cores corresponds to determining a maximum number of containers operable within the cluster of containers as it would necessitate the determination of the maximum number of containers that could run over the cores in order to be maximally parallel. Furthermore, since the workload handling and individual threads of execution of the applications can be analyzed individually corresponds to determining a number of virtual functions and a number of threads operable by each virtual function. Lastly, since the resource scheduler 114 associated with the decisions 151 applied to the OS controls the NIC, this corresponds to the virtual functions being operable with the NIC of the device.);
determining, by the device, a number of cores the daemon is capable to support based on the number of virtual functions, the number of threads, and the maximum number of containers ([0101] Additional processing elements 25, such as emulated kernel services 44, kernel-space parallel processing 52 and user-space buffers 48, may be loaded into user-space 17 and/or kernel space 19 of main memory 18, and/or otherwise made available for processing in one or more of the cores of at least one multi-core processor, such as core 0 multi-core processor 12, to substantially improve processing performance and processing time of software applications, software application groups, and containers running concurrently and/or sequentially under control of the SMP OS and its cores; [0339] Resource scheduler 114, may be in the same or different context with application 87 and PRT 25, decides the distribution of resource to be made available to application 87 and/or PRT 25 and/or other modules, such as buffers 48, in application group 24. Please note that the resource scheduler 114 deciding the distribution of resources to be made available to applications 87 in order to improve processing performance of containers running concurrently under control of the SMP OS and its cores corresponds to Applicant’s determining a number of cores the daemon, i.e., the SMP OS, is capable to support based on the number of virtual functions, number of threads, and maximum number of containers, as these values are used by the resource scheduler 114 in consideration of resource distribution to be made available to applications, in this case, determining the number of cores the SMP OS can support for the containers.);
and establishing, by the device, based on a comparison of the determined number of cores the daemon is capable to support with a number of cores accessible by the device, the cluster of containers ([0369] PRT 25, scheduler 114 and application 93 all run on core 98 and therefore must share the available CPU cycles, e.g. of core 98. Thus, it is desirable to achieve a balance between the resource consumption of scheduler 114, PRT 25 and application 93 to maximize the performance of application 93. The use of groups of programs, related by their types of resource consumption such as groups or containers 22, 24 and 26, and PRT 25 substantially reduces the resource consumption of application 93. Please note that balancing resource consumption to maximize the performance of application 93 in containers by sharing available CPU cycles of core 98 corresponds to establishing the cluster of containers based on a comparison of the determined number of cores the daemon is capable to support with a number of cores accessible by the device, as even if the application 93 running in the established cluster of containers 22, 24, and 26 could be supported further by the capability of the scheduler 114, this is compared to the number of cores accessible to the device, i.e., the available CPU cycles of core 98, which in turn affects the establishment of the containers to balance resource consumption.).
Regarding Claim 19, Sivakumar discloses A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors ([0164] an example computing device (e.g., host), according to techniques described in this disclosure. […] Computing device 800 includes in this example, a bus 242 coupling hardware components of a computing device 800 hardware environment. Bus 242 couples network interface card (NIC) 230, storage disk 246, and one or more microprocessors 210 (hereinafter, “microprocessor 810”). […] A front-side bus may in some cases couple microprocessor 810 and memory device 244.; [0363] In some examples, the computer-readable storage media may comprise non-transitory media.; [0364] The code or instructions may be software and/or firmware executed by processing circuitry including one or more processors. Please note the storage disk 247 of the computing device 800 that is non-transitory and stores instructions for the software that are executed by processing circuitry including one or more processors corresponds to Applicant’s non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to carry out the operations.) to:
maintain a cluster of containers in a user space separate from a kernel space of a device, each container in the cluster of containers to execute a respective one of a plurality of virtual functions ([0011] In aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in an underlying environment, e.g., a Linux-based environment. cRPD may be executed in the user space of the host as a containerized process.; [0071] Containers, including those implementing containerized routing protocol daemons 24, may be deployed to a virtualization environment using a cluster-based framework. Please note that containers deployed to a virtualization environment using a cluster-based framework, where the containers are executing a cRPD in the user space, corresponds to maintaining a cluster of containers in a user space separate from a kernel space of the device, where each container in the cluster executes a respective one of a plurality of virtual functions. Since each container runs a cRPD process that performs respective processing in a virtualized environment, this corresponds to executing respective virtual functions.),
for a network interface card of the device, configured to cause packets received by the device to bypass the kernel space ([0019] The cRPD also programs the data plane on each compute node. For better network packet I/O performance, the DU application may run in the application Pod to bypass the kernel networking stack and abstractions, and thereby use, e.g., zero-copy mechanisms to directly send/receive packets from the physical NIC. Please note that directly receiving packets from the physical NIC by bypassing the kernel networking stack corresponds to Applicant’s causing packets received by the device to bypass the kernel space for the NIC of the device.);
forward, via a load balancing technique, a packet received by the device to a container in the cluster of containers in the user space that executes a virtual function of the plurality of virtual functions ([0086] Orchestrator 50 orchestrates pods comprising container workloads. CNI 312 configures virtual interfaces between pods and the data plane, which may be DPDK-based vRouter 206A. […] In some examples, vRouter 206A has a bonded interface to NIC 321B, which may be an Intel-based NIC that supports DPDK. Bonded interfaces facilitate packet load balancing among fabric interfaces.; [0179] In a DPDK-based deployment of virtual router 206A (shown in FIG. 2), virtual router 206A is installed as a user space 245 application that is linked to the DPDK library. This may lead to faster performance than a kernel-based deployment, particularly in the presence of high packet rates. […] This includes packet polling, packet processing, and packet forwarding. In other words, user packet processing steps are performed by the virtual router 206A DPDK data plane. Please note that the CNI configuring virtual interfaces between pods, comprising container workloads, and the data plane which is DPDK-based vRouter 206A, which has a bonded interface that facilitates packet load balancing, where the virtual router 206A DPDK data plane performs packet processing steps, corresponds to Applicant’s forwarding a packet received by the device to a container in the cluster of containers in the user space that executes a virtual function of the plurality of virtual functions via a load balancing technique. This is because the virtual router 206A DPDK data plane corresponds to the load balancing technique, and forwards the packets received by the device to the respective containers in the pod corresponding to cluster of containers via the virtual interface.);
Sivakumar does not explicitly disclose identify a number of processors accessible to the device to process packets received by the device;
determine, responsive to the number of processors greater than a threshold, to establish a predetermined number of containers for the plurality of virtual functions;
and configure, based on the number of processors, each of the plurality of virtual functions with a predetermined number of queues, wherein each queue of the predetermined number of queues maps to a processor of the number of processors;
and update a queue for a processor managed by the virtual function to cause the processor to process the packet in accordance with the queue.
However, Ho discloses identify a number of processors accessible to the device to process packets received by the device ([0300] applications of group 22, container 91 an/or multithreaded application 93 may be processing workloads generated from clients or server applications—using the OS managed processer and hardware resources (e.g., CPU/core cycles, memories, and network and I/O ports/interfaces)—to produce useful results. For each “unit of workload” (henceforth, shortened to “workload”), an application needs to process to produce results, and as incoming workloads get assigned to an application on an ongoing basis, this processing can be modeled and may be implemented as a queue of workloads in a software processing queue, such as workload processing queues 107 illustrated in SMP OS kernel 46. In workload processing queues 107, first in, first out (FIFO) queues, such as […] packet queues 73 […] may be continually being emptied by the application (such as applications of group 22, container 91 and/or 93). Please note that the CPU cycles of the OS managed processor that is available for workload processing queues 107 such as packet queues 73 corresponds to identifying a number of processors accessible to the device to process packets received by the device.);
determine, responsive to the number of processors greater than a threshold, to establish a predetermined number of containers for the plurality of virtual functions ([0292] for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […] These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Please note that an aggregated threshold with multiple queue thresholds for multiple software processing queues based on CPU utilization corresponds to determining to establish a predetermined number of containers for the plurality of virtual functions responsive to the number of processors greater than a threshold, as each container processes a respective workload that has a software processing queue, and therefore the available CPU cycles being higher, in effect being greater than the queue threshold, would result in a proportional number of containers being established to operate using those CPU cycles.);
and configure, based on the number of processors, each of the plurality of virtual functions with a predetermined number of queues, wherein each queue of the predetermined number of queues maps to a processor of the number of processors ([0290] In this model, each application's thread of execution is continually processing workloads (per their abstractions, representations, and data in the queues) from parallel queues to produce results, operating within the constraints of the resources (e.g., CPU/cores, memory, and storage, etc.) assigned to it either dynamically or statically. Please note that the thread of execution of each application operating parallel queues within the constraints of the CPU dynamically assigned to it corresponds to configuring each of the plurality of virtual functions with a predetermined number of queues where each queue of the predetermined number of queues maps to a processor of the number of processors based on the number of processors. This is because the number of queues of each application is configured based on the number of processors, i.e., the resource constraints, and each application corresponds to each of the plurality of virtual functions and is mapped to the processor that is executing it.);
and update a queue for a processor managed by the virtual function to cause the processor to process the packet in accordance with the queue ([0396] Referring now to FIG. 17, as illustrated in workload tuning operation 163, scheduler 114 may cause PRT 25 to schedule or reschedule data transfers with various different software processing queues in queues 82 in accordance with dynamic workload changes, e.g. during processing of application 93 by core 99 […] data between the software queues and its relevant external entities (e.g., hardware queues of input/output packets in network interface cards). Please note that software processing queues operating with hardware queues of input/output packets being scheduled by scheduler 114 during processing of application 93 by core 99 corresponds to updating a queue to cause processing of the packet in accordance with the queue, and since the queue is run on a particular core of the cores managed by the system, this corresponds to the queue being for the processor and the processor processing the packet.).
Sivakumar and Ho are both considered to be analogous to the claimed invention because they are in the same field of containerized computer packet processing. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Sivakumar to incorporate the teachings of Ho to modify the system maintaining container clusters in a user space separate from the kernel space, each container executing respective virtual functions, and forwarding packets received by the device to containers via a load balancing technique to identify a number of processors accessible to the device to process packets received by the device, determine to establish a predetermined number of containers for the virtual functions if the number of cores is greater than a threshold, configure each of the virtual functions with a predetermined number of queues based on the number of processors, wherein each queue of the predetermined number of queues maps to a processor of the plurality of processors, and update a queue for a core of the cores managed by the virtual function to cause it to process the packet, allowing for improved packet processing performance and resource usage efficiency, as described in Ho.
Regarding Claim 21, Sivakumar-Ho as described in Claim 11, Ho further discloses wherein the threshold is a threshold representing a number of cores ([0292] Compute and configure software processing queues' queue thresholds: for each of the above workload- and application-specific queue, construct and compute a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration […]These constitute a processing queue threshold. Thresholds can be one for each software processing queue, or an aggregated one computed as a function of multiple queue thresholds for multiple software processing queues. Queue threshold can also be configured manually, instead of automatically via statistical analysis of measured data, etc.
Please note that the thresholds for software processing queues being based on CPU core utilization corresponds to Applicant’s threshold representing a number of cores, as in order to determine the CPU core utilization that informs the computing of the QoE/QoS threshold, it is inherently necessary to have an indication of a number of cores of the device, represented in the resultant threshold.).
Response to Arguments
Applicant's arguments filed 04/01/2026 have been fully considered but they are not persuasive.
Applicant’s arguments are summarized as follows:
Since Claim 20 has been canceled, its antecedent basis issue under 35 U.S.C. 112 is resolved.
Regarding the rejection of independent Claim 1 under 35 U.S.C. 103, the references, alone or in combination, do not teach identifying a number of accessible cores or comparing that number to a threshold. This is because the QoE/QoS threshold of Ho is not a threshold of available cores, but rather a metric used to indicate workload congestion for a specific software application. Furthermore, the broadest reasonable interpretation of the claimed threshold does not stretch to include the QoE/QoS threshold in the Ho reference, since the claims, as written, require that a number of available cores is compared to the threshold; thus, a reasonable interpretation is that the threshold relates to available cores. It is unreasonable to equate it to the workload congestion QoE/QoS threshold as there is no reasonable way to compare a number of available cores to a measurement of workload congestion. Sivakumar also does not disclose determining whether a number of available cores is greater than a threshold. Therefore, the references, alone or in combination, do not teach the elements of Claim 1, and its rejection should be withdrawn.
Since independent Claims 13 and 19 contain similar elements to allowable independent Claim 1, their rejections under 35 U.S.C. should also be withdrawn.
Since the dependent claims depend on allowable independent Claims, their rejections under 35 U.S.C. should also be withdrawn.
Regarding A, since Claim 20 has been cancelled, and the limitation previously lacking antecedent basis has been corrected once it was incorporated into Claim 19, the amendment is sufficient to overcome the antecedent basis rejection under 35 U.S.C. 112. The rejection is withdrawn.
Regarding B, the examiner respectfully disagrees. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., the threshold being a threshold for a number of available cores) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, contrary to Applicant’s argument that the broadest reasonable interpretation of the threshold does not include the QoE/QoS threshold of Ho, the examiner respectfully disagrees. As stated above in [0292], “a workload-congestion indicative QoE/QoS threshold, for example, as a function of (a) the average queue length of the application, measured while “saturating” the CPU utilization or CPU core utilization on which the application or application's process/thread runs over a set duration.” Therefore, as the threshold of Ho is determined as a function of saturating the CPU core utilization, it necessarily involves the number of CPU cores being utilized. Therefore, the recited features can be found in the cited combination of references, and independent Claim 1 remains rejected under 35 U.S.C. 103 for the reasons stated above, and the combinations cited would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the application. The rejections under 35 U.S.C. 103 are maintained.
Regarding C, the examiner respectfully disagrees. Contrary to Applicant’s arguments, because the independent Claims 13 and 19 contain similar limitations to rejected Claim 1 and do not add limitations that overcome the rejection, they likewise remain rejected. The rejections under 35 U.S.C. 103 are maintained.
Regarding D, the examiner respectfully disagrees. Contrary to Applicant’s arguments, because the dependent claims depend on unpatentable independent Claims 1, 13, and 19 and do not add limitations that overcome the rejection, they likewise remain rejected. The rejections under 35 U.S.C. 103 are maintained.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mariappan et al. (US-20220278927-A1) discloses creating virtual networks to enable packetized communication using containers, bypassing the kernel to exchange packets in the user space, executing applications in a virtualized environment, and performing packet operations enabled by a NIC (see [0008-0010, 0034, 0041-44]).
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARAZ T AKBARI whose telephone number is (571)272-4166. The examiner can normally be reached Monday-Thursday 9:30am-7:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, April Blair can be reached at (571)270-1014. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FARAZ T AKBARI/Examiner, Art Unit 2196
/APRIL Y BLAIR/Supervisory Patent Examiner, Art Unit 2196
Prosecution Insights