DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
1. Claims 9 and 22 are currently amended.
2. Claims 9-22 are pending.
3. Claims 9-22 are rejected.
Response to Arguments
4. Regarding Prior Art Rejections:
The arguments regarding the rejections under 35 U.S.C. § 102(a)(1) and 35 U.S.C. § 103 challenge certain limitations. These limitations are newly added and were therefore not addressed in the previous rejection; therefore, the arguments are moot. The amendments are newly addressed by the new grounds of rejection under 35 U.S.C. § 103.
Applicant’s arguments with respect to claim(s) 9 and 22 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Specifically, Gotsis et al. is no longer applied and applicant’s remarks solely address Gotsis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 9, 12-13, and 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Abali et al. US 20150169337 A1 in view of Wang et al. US 20200042343 A1, in view of Simionescu et al. US 20210089454 A1, and in further view of Sampathkumar US 20190188156 A1.
6. With regard to claim 9, Abali teaches:
An apparatus comprising a processor ([0022] The host computer 200 can include at least one processor) and a non-transitory, computer-readable memory comprising instructions which, when executed by the processor, cause the processor to ([0028] In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.):
determine a network throughput for migrating a virtual machine (VM) from a source site to a target site ([0008] VM mobility cost is the cost to move the VM to the target host server computer; [0008] The VM mobility cost is generally influenced both by available network bandwidth and the workload writes to the VM memory which must be transferred across the network to the target host server computer during VM relocation; [0008] Dirty pages must be transferred from source server to the target server during VM relocation; [0010] In yet another aspect of the embodiment, the VM mobility cost even yet further includes an expected throughput determined according to a network topology identified for the data communications network; Examiner’s Note: Mobility cost is defined as relocation cost to move the VM from the source server (source site) to the target host server computer (target site). One factor in determining mobility costs is network bandwidth or expected throughput according to a network topology, which is analogous with a network throughput for migrating a VM from a source site to a target site.);
determine an available processing capacity for migrating the VM ([0008] Rather, contemporary VM relocation tools only relocate VMs based on VM running cost and system status including memory capacity, CPU utilization, energy consumption, and given policy; Examiner’s Note: Another factor of relocation is CPU utilization, which is analogous with an available processing capacity for migrating the VM.);
determine an expected level of dirtiness of a memory of the VM ([0010] In another aspect of the embodiment, the VM mobility cost additionally includes an expected page dirtying rate for each of the VMs; Examiner’s Note: Another factor of relocation is expected page dirtying rate, which is analogous with an expected level of dirtiness of a memory of the VM.);
Abali does not explicitly teach based on the network throughput, the available processing capacity, and the expected level of dirtiness of the memory of the VM, determine a granularity level for tracking the memory of the VM; track portions of the memory of the VM for dirtiness, wherein a size of the portions is based on the granularity level; migrating the VM from the source site to the target site; and dynamically adjust the granularity level during migration by reducing the granularity level through marking progressively larger groups of contiguous sub-pages as dirty to reduce tracking overhead based on detected patterns of contiguous dirtiness.
However, in analogous art, Wang teaches:
based on the network throughput, the available processing capacity, and the expected level of dirtiness of the memory of the VM, determine a granularity level for tracking the memory of the VM ([0019] To enable efficient coarse-grained or fine-grained tracking of modified data, a limited amount of cache or memory space can be allocated to coarse and fine-grained modified page trackers. In some examples, in an event that insufficient cache or memory space are available for entries in the coarse modified page tracker, one or more entries can overflow to another memory or storage region that keeps track of checkpoint data changes at a coarse-level granularity of the same or lower level of granularity (e.g., page or larger in size) than that tracked by the coarse modified page tracker in cache or memory space. In some examples, an entry can include a single bit to indicate whether a fine-grained or coarse-grained region has been changed. For example, if a page is 4 kilobytes and any checkpoint data in the page has changed, a single bit in an entry in a coarse-grained modified page tracker can identify the page has changed (e.g., dirtied). In another example, if a cache line worth of checkpoint data has changed, an entry in the fine-grained modified page tracker that tracks changes to the cache line level can indicate the cache line has been changed (e.g., dirtied); [0025] After checkpoint replication occurs, monitoring of both the network and memory traffic can occur using profiling tool such as VTune. The profiling results may show the memory and network traffic of the first benchmark is significantly smaller (e.g., 1/64) of those of the second benchmark. However, using a coarse-grained, 4 KB-page-level, modified tracker will write the same amount of data to the checkpoint in both benchmarks. A fine-grained, 64-Byte cache line, modified tracker will store 1/64.sup.th of the data in the first benchmark compared to the second benchmark; Examiner’s Note: Monitored memory and network traffic (network throughput), cache or memory space allocation (processing capacity), and tracking of dirty/changed checkpoint data (expected level of dirtiness) are all factors in determining granularity level.); and
track portions of the memory of the VM for dirtiness, wherein a size of the portions is based on the granularity level ([0031] According to some embodiments, memory controller 204 can use a checkpoint change tracker 206 to identify changes made at fine-grain or coarser levels of a memory region. For example, with respect to checkpoint data 214, checkpoint change tracker 206 can identify changes made to checkpoint data 214 at a fine-grain or coarser levels. For example, a fine-grain level can be at a cache line level (e.g., 32 bytes, 64 bytes, 128 bytes, or other multiples of 16 bytes or other numbers of bytes). A coarse-grain level can be page sized (e.g., 2 kilobytes, 4kilobytes, 8 kilobytes, 16 kilobytes, or other multiples of 2 kilobytes, or other sizes).).
migrate the VM from the source site to the target site ([0001] Various examples described herein relate to virtual machine migration; [0013] Checkpoint data can be a memory image of a VM or application at some time of execution. All changes in memory (e.g., cache and memory) by the VM or application are captured. In the VM context, a snapshot or checkpoint is a consistent full representation of the VM. A snapshot or checkpoint can be used to suspend a VM so that it can later be restored on the same platform, or migrated to a secondary platform; [0075] As part of a checkpoint replication procedure, the fine-grained tracker cache, coarse level tracker cache, and coarse level tracker in memory are checked by a virtual machine manager (VMM), hypervisor, orchestrator, or other software to determine what memory addressable parts of the checkpoint data has changed. The checkpoint data with the corresponding region modification indicators set any tracker are read out from the memory device and sent over the network to a destination memory device associated with a replica VM or migrated VM.); and
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali with the teachings of Wang where based on the network throughput, the available processing capacity, and the expected level of dirtiness of the memory of the VM, determine a granularity level for tracking the memory of the VM; track portions of the memory of the VM for dirtiness, wherein a size of the portions is based on the granularity level; migrating the VM from the source site to the target site; and dynamically adjust the granularity level during migration by reducing the granularity level to reduce tracking overhead. As Abali mentioned, migration cost to relocation, or migrate, VMs is based on the network throughput, the available processing capacity, and the expected level of dirtiness of the memory of the VM. To lower migration cost, Wang has determined a granularity level that is based on these three factors. This enables the system to determine the ideal granularity level to lower migration cost and allow for efficient VM migration, as discussed in Wang ([0018]). Once these factors are determined, the VM is migrated from a source site to a destination site.
Abali and Wang fail to explicitly teach dynamically adjust the granularity level during migration by reducing the granularity level through marking progressively larger groups of contiguous sub-pages as dirty to reduce tracking overhead based on detected patterns of contiguous dirtiness.
However, in analogous art, Simionescu teaches:
dynamically adjust the granularity level during migration by reducing the granularity level through marking progressively larger groups of contiguous sub-pages as dirty to reduce tracking overhead based on detected patterns of contiguous dirtiness ([0016] In an illustrative example, the DRAM can be structured as a cache that stores recently accessed, and/or highly accessed data so that such data can be accessed quickly by the host system. The DRAM data cache can be partitioned into two different caches that are managed at different data sizes. One of the partitions can include a page cache utilizing a larger granularity (i.e., a larger size) and the second partition can include a sector cache utilizing a smaller granularity (i.e., a smaller size). Because the page cache utilizes a larger data size, less metadata is used to manage the data (e.g., only a single valid bit for the entire page). The smaller data size of the sector cache can use larger amounts of metadata (e.g., larger number of valid and dirty bits along with tags etc.) but can allow for host access data to be tracked with more detail, thus increasing the overall cache hit rate in the DRAM data cache. Increasing the hit rate in the DRAM data cache can provide performance comparable to a DIMM with only DRAM memory devices but can additionally provide larger capacity memory, lower cost, and support for persistent memory. In addition, tracking data at a sector granularity can reduce bandwidth utilization between the DRAM memory device and the 3D cross-point memory when filling missing data is the sector cache or writing back dirty data from the DRAM memory device to the 3D cross-point memory. In some embodiments, the controller can track access statistics of segments of data having the smaller granularity stored at the DRAM memory device. Based on the access statistics, the controller can update the segment of data to the larger granularity by retrieving additional data associated with the segment of data and forming a new segment, at the larger granularity, which includes the segment of data and the additional data. With the larger granularity, less metadata is used to represent the segment of data. Therefore, less processing overhead is incurred when managing a frequently accessed segment, for which the metadata is updated often; Examiner’s Note: Simionescu’s larger granularity is analogous with reducing granularity because the page cache it utilizing a larger, coarse-grained size, which saves metadata space.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali and Wang with the teachings of Simionescu to dynamically adjust the granularity level during migration by reducing the granularity level to reduce tracking overhead. As Abali mentioned, migration cost to relocation, or migrate, VMs is based on the network throughput, the available processing capacity, and the expected level of dirtiness of the memory of the VM. To lower migration cost, Wang has determined a granularity level that is based on these three factors. This enables the system to determine the ideal granularity level to lower migration cost and allow for efficient VM migration. Similarly, Simionescu teaches of dynamically reducing the granularity level in order to reduce tracking overhead. By updating the segment of data to a larger granularity, less metadata is used to represent the segment of data. Therefore, less processing overhead is incurred, as discussed in Simionescu ([0016]).
Although Simionescu teaches of dynamically adjusting the granularity level during migration by reducing the granularity level to reduce tracking overhead, Simionescu fails to explicitly teach through marking progressively larger groups of contiguous sub-pages as dirty to reduce tracking overhead based on detected patterns of contiguous dirtiness.
However, in analogous art, Sampathkumar teaches:
dynamically adjust the granularity level during migration by reducing the granularity level through marking progressively larger groups of contiguous sub-pages as dirty to reduce tracking overhead based on detected patterns of contiguous dirtiness ([0023] The flushing manager 108 uses the data structure 110 to identify contiguous groupings of dirty data blocks. When a flush is triggered, the flushing manager 108 reads the first cache window referenced by the data structure 110. If all cache blocks of the cache window are dirty, then the flushing manager attempts to build a sequence of dirty cache blocks (e.g., sequence as allocated to the disc array 106) using the dirty cache blocks of the cache window. If a sequence already exists (e.g., the flushing manager already started building a sequence) and the current window is adjacent to the previously started sequence, then the dirty blocks of the cache window are concatenated or added to the sequence; [0028] The above described implementations are utilized to improve the storage system by constructing a sequence of dirty cache blocks that are aligned and coalesced within a data stripe in the disc array 106.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, and Simionescu with the teachings of Sampathkumar through marking progressively larger groups of contiguous sub-pages as dirty to reduce tracking overhead based on detected patterns of contiguous dirtiness. Similarly to Simionescu, Sampathkumar also aims to reduce tracking overhead. Sampathkumar does this by marking progressively larger groups of continuous sub-pages as dirty. This is reflected in Sampathkumar’s process of building a sequence of dirty cache blocks. This is done in order to improve the storage system by constructing a sequence of dirty cache blocks that are aligned and coalesced within a data stripe in the disc array; therefore, reducing tracking overhead. As discussed in Sampathkumar, groups or segments of logically sequential data may be separated into different physical storage devices such as to improve the speed of I/O operations. For example, if an I/O operation is directed to a segment of logically sequential addresses and the logically sequential addresses are stored on a single physical storage device, then the single device takes time to process the full I/O operation. In contrast, if the segment of logically sequential addresses is stored across a plurality of physical storage devices, then the processing is allocated between the separate physical storage devices, which improves I/O performance ([0012]).
7. With regard to claim 12, Wang further teaches:
wherein determining the granularity level comprises:
migrating a first subset of the memory of the VM, wherein migrating the first subset comprises tracking a dirtiness of first portions of the first subset, wherein a size of the first portions is based on a first preliminary granularity level ([0012] Checkpoint replication is used to ensure virtual machine availability in the case of a system crash or power failure, and also for rapid migration of live and active VMs. During a checkpoint replication procedure, the checkpoint data is read from a host VM and sent over a network or fabric (or other communication medium) to a backup computing platform machine. If a system fails, a checkpoint can be used to start a VM or application at its checkpoint to avoid starting from the VM or application from its beginning and avoid incurring latency as a result. Maintaining backup VMs can reduce loss of a VM's completed work caused by failure of a primary VM; [0037] [0037] For example, a virtual machine manager (VMM) (e.g., a VMM executed by computing platform 200-1) or other software or hardware can cause the transmission of merely the checkpoint data 214 corresponding to changed regions of memory at a fine-grained or coarse-grained level. [...] In addition, if a coarse-grained level region is 4096 bytes, then any of checkpoint data 214 corresponding to coarse level memory region that is identified as changed is transmitted to another device (e.g., computing platform 200-1 or 200-N). Accordingly, as part of a checkpoint period in this example, a mix of zero or more 64 byte segments (or other sizes) and zero or more 4096 byte segments (or other sizes) including portions of checkpoint data 214 are transmitted to another device. A processor that executes a VMM or other process can merge checkpoint data changes with other checkpoint data that has not changed since a prior checkpoint operation. Composite checkpoint data including changed and unchanged checkpoint data can be used by a VM for backup or VM migration or container backup or migration. In some examples, checkpoint data can be used for migration, replication, or backup of VMs or containers; [0059] A memory controller used by a CPU and/or a memory device can receive the write request. Checkpoints could be initiated by a VM at an appropriate time in its execution, or could be initiated periodically by the system. When a checkpoint is initiated, the tracked modified data from the sparse and dense trackers are stored. Tracking modified data is based on whether the address belongs to a tracked memory region.);
determining a first performance metric associated with migrating the first subset ([0014] To save space, VMMs monitor specific memory regions that contain data from the VMs that need to be checkpointed. For the monitored memory regions, only data that has been modified since the last checkpoint needs to be stored. Checkpoints could be used to store the state of the whole system, or could be used to store the state for only one or a few VMs; [0023] Tracking updates to a memory regions of interest is required for a variety of usage models. Various embodiments can reduce the resources required to track and process such memory modifications for use cases that do not require full checkpoints, including usages which may not require storage or transmission of the modifications.);
migrating a second subset of the memory of the VM, wherein migrating the second subset comprises tracking a dirtiness of second portions of the second subset, and wherein a size of the second portions is based on a second preliminary granularity level ([0037] Composite checkpoint data including changed and unchanged checkpoint data can be used by a VM for backup or VM migration or container backup or migration. In some examples, checkpoint data can be used for migration, replication, or backup of VMs or containers; [0059] A memory controller used by a CPU and/or a memory device can receive the write request. Checkpoints could be initiated by a VM at an appropriate time in its execution, or could be initiated periodically by the system. When a checkpoint is initiated, the tracked modified data from the sparse and dense trackers are stored. Tracking modified data is based on whether the address belongs to a tracked memory region; [0074] At 444, an entry is added to the coarse level tracker cache and the added entry is updated to indicate that a modification has been made to the address associated with the write request. For example, adding a new entry can include adding a new entry to the coarse level tracker cache or using an invalid entry (after clearing its indicators that any modification has been made). A region modification indicator can be set to indicate the address associated with the write request is modified. A tag for the entry can be set as significant bytes of the address associated with the write request. A valid indicator can be set for the entry to indicate the entry is valid; [0075] As part of a checkpoint replication procedure, the fine-grained tracker cache, coarse level tracker cache, and coarse level tracker in memory are checked by a virtual machine manager (VMM), hypervisor, orchestrator, or other software to determine what memory addressable parts of the checkpoint data has changed. The checkpoint data with the corresponding region modification indicators set any tracker are read out from the memory device and sent over the network to a destination memory device associated with a replica VM or migrated VM.);
determining a second performance metric associated with migrating the second subset ([0014] To save space, VMMs monitor specific memory regions that contain data from the VMs that need to be checkpointed. For the monitored memory regions, only data that has been modified since the last checkpoint needs to be stored. Checkpoints could be used to store the state of the whole system, or could be used to store the state for only one or a few VMs; [0023] Tracking updates to a memory regions of interest is required for a variety of usage models. Various embodiments can reduce the resources required to track and process such memory modifications for use cases that do not require full checkpoints, including usages which may not require storage or transmission of the modifications.); and
based on the first performance metric and the second performance metric, determining the granularity level ([0025] The following provides an example of comparison among approaches to track updates to checkpoint data. In a first benchmark, a super page of checkpoint data (e.g., 2 MB) is allocated. Write requests are issued to each of the 4 KB pages. A second benchmark allocates a super page of checkpoint data (e.g. 2 MB). Write requests are issued to every single 64 B block in the whole super page. After checkpoint replication occurs, monitoring of both the network and memory traffic can occur using profiling tool such as VTune. The profiling results may show the memory and network traffic of the first benchmark is significantly smaller (e.g., 1/64) of those of the second benchmark. However, using a coarse-grained, 4 KB-page-level, modified tracker will write the same amount of data to the checkpoint in both benchmarks. A fine-grained, 64-Byte cache line, modified tracker will store 1/64.sup.th of the data in the first benchmark compared to the second benchmark; [0032] Checkpoint change tracker 206 can use coarse-grained change tracker 208 to track changes to checkpoint data 214 at a coarse level. Checkpoint change tracker 206 can use fine-grained change tracker 210 to track changes to checkpoint data 214 at a fine-grained level. In some examples, checkpoint change tracker 206 tracks any received requested changes (e.g., writes) to a region of memory 212 that stores a portion of checkpoint data 214. Checkpoint change tracker 206 determines if an entry in coarse-grained change tracker 208 is present that indicates that a coarse level region of memory that stores checkpoint data 214 has been indicated as changed since a completion of a prior checkpoint period or procedure. If such entry is present and valid in coarse-grained change tracker 208, the change has already been accounted for at a coarse level. If such entry is not present or is not valid, then changes to a region of memory that stores checkpoint data 214 are tracked at a fine-grained change tracker using fine-grained change tracker 210.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali with the teachings of Wang wherein determining the granularity level comprises: migrating a first subset of the memory of the VM, wherein migrating the first subset comprises tracking a dirtiness of first portions of the first subset, wherein a size of the first portions is based on a first preliminary granularity level; determining a first performance metric associated with migrating the first subset; migrating a second subset of the memory of the VM, wherein migrating the second subset comprises tracking a dirtiness of second portions of the second subset, and wherein a size of the second portions is based on a second preliminary granularity level; determining a second performance metric associated with migrating the second subset; and based on the first performance metric and the second performance metric, determining the granularity level. By monitoring performance metrics to determine a granularity level, memory and network system performance can be improved. Specifically in Wang, reducing the memory and network traffic caused by checkpoint replication can significantly improve memory and network system performance and reduce associated power consumption. To save space, VMMs monitor specific memory regions that contain data from the VMs that need to be checkpointed. For the monitored memory regions, only data that has been modified since the last checkpoint needs to be stored. Checkpoints could be used to store the state of the whole system, or could be used to store the state for only one or a few VMs ([0014]).
8. With regard to claim 13, Wang further teaches:
wherein determining the granularity level comprises selecting one of the first preliminary granularity level or the second preliminary granularity level ([0018] Various embodiments track modified content in a virtual machine's checkpoint data in either fine or coarse granularity-level (or both; [0018] Due to fine-grained tracking, various embodiments can avoid copying a large portion of clean or unmodified checkpoint data, and therefore significantly reduce the redundant memory and network traffic and reduce power consumption. Fine-grained tracking saves checkpoint construction and reconstruction latency and processing, and also saves the space needed to store the checkpoint compared to coarse-grain tracking; Examiners’ Note: Two granularity level options are presented, either fine-grained tracking or coarse-grained tracking. In this case, fine-grained tracking is chosen due to its ability to reduce redundant memory and network traffic and reduce power consumption.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali with the teachings of Wang wherein determining the granularity level comprises selecting one of the first preliminary granularity level or the second preliminary granularity level in order to choose the most ideal granularity level for efficient VM migration. In Wang, it is mentioned that fine-grained tracking was chosen since it was able to significantly reduce the redundant memory and network traffic and reduce power consumption, as discussed in Wang ([0018]).
9. With regard to claim 19, Simionescu further teaches:
wherein the instructions further cause the processor to:
update the granularity level based on tracking portions of the memory of the VM for dirtiness ([0016] One of the partitions can include a page cache utilizing a larger granularity (i.e., a larger size) and the second partition can include a sector cache utilizing a smaller granularity (i.e., a smaller size). Because the page cache utilizes a larger data size, less metadata is used to manage the data (e.g., only a single valid bit for the entire page). The smaller data size of the sector cache can use larger amounts of metadata (e.g., larger number of valid and dirty bits along with tags etc.) but can allow for host access data to be tracked with more detail, thus increasing the overall cache hit rate in the DRAM data cache. Increasing the hit rate in the DRAM data cache can provide performance comparable to a DIMM with only DRAM memory devices but can additionally provide larger capacity memory, lower cost, and support for persistent memory. [...] Based on the access statistics, the controller can update the segment of data to the larger granularity by retrieving additional data associated with the segment of data and forming a new segment, at the larger granularity, which includes the segment of data and the additional data. With the larger granularity, less metadata is used to represent the segment of data. Therefore, less processing overhead is incurred when managing a frequently accessed segment, for which the metadata is updated often.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali and Wang with the teachings of Simionescu to dynamically adjust the granularity level during migration by reducing the granularity level to reduce tracking overhead. As Abali mentioned, migration cost to relocation, or migrate, VMs is based on the network throughput, the available processing capacity, and the expected level of dirtiness of the memory of the VM. To lower migration cost, Wang has determined a granularity level that is based on these three factors. This enables the system to determine the ideal granularity level to lower migration cost and allow for efficient VM migration. Similarly, Simionescu teaches of dynamically reducing the granularity level in order to reduce tracking overhead. By updating the segment of data to a larger granularity, less metadata is used to represent the segment of data. Therefore, less processing overhead is incurred, as discussed in Simionescu ([0016]).
10. With regard to claim 20, Wang further teaches:
wherein updating the granularity level comprises:
determining a number of sub-pages of a page of the VM that are dirtied ([0018] A coarse modified page tracker can be used by a memory controller to keep track of the dirty or changed checkpoint data at a coarse granularity (e.g., page or other size) since a prior checkpoint procedure was performed (e.g., started or completed); Examiner’s Note: Pages of a VM are being tracked for dirtiness at a coarse level.); and
based on the number of dirtied sub-pages exceeding a threshold, reducing the granularity level ([0018] A coarse modified page tracker can be used by a memory controller to keep track of the dirty or changed checkpoint data at a coarse granularity (e.g., page or other size) since a prior checkpoint procedure was performed (e.g., started or completed); [0019] To enable efficient coarse-grained or fine-grained tracking of modified data, a limited amount of cache or memory space can be allocated to coarse and fine-grained modified page trackers. In some examples, in an event that insufficient cache or memory space are available for entries in the coarse modified page tracker, one or more entries can overflow to another memory or storage region that keeps track of checkpoint data changes at a coarse-level granularity of the same or lower level of granularity (e.g., page or larger in size) than that tracked by the coarse modified page tracker in cache or memory space. In some examples, an entry can include a single bit to indicate whether a fine-grained or coarse-grained region has been changed; Examiner’s Note: Pages are being tracked at a coarse level for dirtiness. If a number of pages to be read (sub-pages) exceeds the allocated amount of cache or memory space (threshold exceeded), then some pages are overflowed to another memory or storage region that tracks data at the same or lower level of granularity.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali with the teachings of Wang wherein updating the granularity level comprises: determining a number of sub-pages of a page of the VM that are dirtied; and based on the number of dirtied sub-pages exceeding a threshold, reducing the granularity level. In the case of a cache/memory space being insufficient, pages are able to be overflowed to other memory regions and continue being monitored. Therefore, enabling the continued execution of a VM and its processes, as discussed in Wang ([0019]).
11. With regard to claim 21, Wang further teaches:
wherein updating the granularity level comprises:
determining a number of contiguous sub-pages of a page of the VM that are dirtied ([0019] For example, if a page is 4 kilobytes and any checkpoint data in the page has changed, a single bit in an entry in a coarse-grained modified page tracker can identify the page has changed (e.g., dirtied); [0045] When modified bits 304A is 8 bytes (i.e., 64 bits) in length, modified bits 304A can track modification (or non-modification) of up to 64 adjacent pages starting at a physical address indicated in part by tag 304B; Examiner’s Note: Another word for dirtied is changed or modified. Pages of a VM are being tracked for dirtiness at a coarse level, and the pages can be adjacent (contiguous).); and
based on the number of contiguous dirtied sub-pages exceeding a threshold, reducing the granularity level ([0018]; A coarse modified page tracker can be used by a memory controller to keep track of the dirty or changed checkpoint data at a coarse granularity (e.g., page or other size) since a prior checkpoint procedure was performed (e.g., started or completed); [0019] To enable efficient coarse-grained or fine-grained tracking of modified data, a limited amount of cache or memory space can be allocated to coarse and fine-grained modified page trackers. In some examples, in an event that insufficient cache or memory space are available for entries in the coarse modified page tracker, one or more entries can overflow to another memory or storage region that keeps track of checkpoint data changes at a coarse-level granularity of the same or lower level of granularity (e.g., page or larger in size) than that tracked by the coarse modified page tracker in cache or memory space. In some examples, an entry can include a single bit to indicate whether a fine-grained or coarse-grained region has been changed; Examiner’s Note: Pages are being tracked at a coarse level for dirtiness, and they can be adjacent, or contiguous, as made apparent above. If a number of adjacent pages to be read (sub-pages) exceeds the allocated amount of cache or memory space (threshold exceeded), then some pages are overflowed to another memory or storage region that tracks data at the same or lower level of granularity.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali with the teachings of Wang wherein updating the granularity level comprises: determining a number of contiguous sub-pages of a page of the VM that are dirtied; and based on the number of contiguous dirtied sub-pages exceeding a threshold, reducing the granularity level. In the case of a cache/memory space being insufficient, pages are able to be overflowed to other memory regions and continue being monitored. Additionally, these pages can be monitored adjacently, as shown in Wang ([0045]). Therefore, enabling the continued execution of a VM and its processes, as discussed in Wang ([0019]).
12. Regarding claim 22, it is rejected under the same reasoning as claim 9 above. Therefore, it is rejected under the same rationale.
13. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Abali et al. US 20150169337 A1; Wang et al. US 20200042343 A1; Simionescu et al. US 20210089454 A1; and Sampathkumar US 20190188156 A1, as applied in claim 1, in further view of Dong US 20170329623 A1.
14. With regard to claim 10, Abali, Wang, Simionescu, and Sampathkumar teach the apparatus of claim 9 but fail to explicitly teach wherein determining the expected level of dirtiness of the memory of the VM is based on one or more historical levels of dirtiness of historical migrations of the VM.
However, in analogous art, Dong further teaches:
wherein determining the expected level of dirtiness of the memory of the VM is based on one or more historical levels of dirtiness of historical migrations of the VM ([0057] The VM checkpoint will include all of the dirtied memory pages accumulated during the past epoch, which includes both the dirtied pages tracked using the dirty-tracking scheme associated with frame buffer and command buffer accesses, and the predicted to-be dirtied pages; Examiner’s Note: Dirtied memory pages accumulated during past epoch are used to predict the to-be dirtied pages.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar with the teachings of Dong wherein determining the expected level of dirtiness of the memory of the VM is based on one or more historical levels of dirtiness of historical migrations of the VM in order to more accurately predict to-be dirtied pages by using past dirtied memory page data, as discussed in Dong ([0057]). Therefore, providing more accurate predictions.
15. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Abali et al. US 20150169337 A1; Wang et al. US 20200042343 A1; Simionescu et al. US 20210089454 A1; and Sampathkumar US 20190188156 A1, as applied in claim 1, in further view of Danilov et al. US 9880870 B1.
16. With regard to claim 11, Abali, Wang, Simionescu, and Sampathkumar teach the apparatus of claim 9 but fail to explicitly teach wherein determining the expected level of dirtiness of the memory of the VI comprises: sampling a first portion of the memory of the VM to determine a first portion dirtiness of the first portion; sampling a second portion of the memory of the VM to determine a second portion dirtiness of the second portion; and based on the first portion dirtiness and the second portion dirtiness, calculating the expected level of dirtiness of the memory of the VM.
However, in analogous art, Danilov further teaches:
wherein determining the expected level of dirtiness of the memory of the VM comprises (Col. 15, lines 13-20):
sampling a first portion of the memory of the VM to determine a first portion dirtiness of the first portion (Col. 14, lines 66-67 - Col. 15, lines 1-5, During stage A (element 316), which may be called the “pre-copy” stage, a set of selected dirty memory portions or pages containing state information of the to-be-migrated GVM may be copied to the destination VH while the source GVM remains active. For example, all the dirty memory pages that a hypervisor has identified at a point in time T1 may be copied over; Examiner’s Note: Dirty memory of a first portion/pages is migrated to the destination. This first portion has a level of dirtiness associated with it.);
sampling a second portion of the memory of the VM to determine a second portion dirtiness of the second portion (Col. 15, lines 5-10, However, during this first step, some new portions of memory may be written to, since the pre-migration GVM remains running Thus, at least in some cases, a second iteration of copying may be required, which includes copying a second set of memory portions that were written to during the first iteration; Examiner’s Note: : Dirty memory of a second portion/pages is migrated to the destination. This second portion has a level of dirtiness associated with it.); and
based on the first portion dirtiness and the second portion dirtiness, calculating the expected level of dirtiness of the memory of the VM (Col. 15, lines 13-20, These iterative copying steps may be expected to converge (that is, the amount of memory that has to be copied during a given iteration may in general tend to be smaller than the amount that was copied during a previous iteration) in most operating environments. At some point the combined size of the dirty memory portions to be copied may fall below a threshold, and stage A of the memory copy may be completed; Examiner’s Note: The combined portions of dirty memories that have been copied are expected to fall below a threshold. This threshold indicates an expected level of dirtiness of the memory of the VM.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar with the teachings of Danilov wherein determining the expected level of dirtiness of the memory of the VM comprises sampling a first portion of the memory of the VM to determine a first portion dirtiness of the first portion; sampling a second portion of the memory of the VM to determine a second portion dirtiness of the second portion; and based on the first portion dirtiness and the second portion dirtiness, calculating the expected level of dirtiness of the memory of the VM. This is because these iterative copying steps may be expected to converge (that is, the amount of memory that has to be copied during a given iteration may in general tend to be smaller than the amount that was copied during a previous iteration), as discussed in Danilov (Col. 15, lines 13-20). Therefore, allowing for an accurate calculation of the expected level of dirtiness of the memory of the VM.
17. Claims 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Abali et al. US 20150169337 A1; Wang et al. US 20200042343 A1; Simionescu et al. US 20210089454 A1; and Sampathkumar US 20190188156 A1, as applied in claim 1, in further view of Parker US 20140026133 A1.
18. With regard to claim 14, Abali, Wang, Simionescu, and Sampathkumar teach the apparatus of claim 12 but fails to explicitly teach wherein at least one of the first performance metric and the second performance metric comprises a migration speed.
However, in analogous art, Parker further teaches:
wherein at least one of the first performance metric and the second performance metric comprises a migration speed ([0022] Virtual container server 220 may receive instructions from orchestration server 230 to add, remove, combine, or migrate (i.e., move) virtual machines associated with virtual containers stored by virtual container server 220. In some implementations, virtual container server 220 may provide performance data, associated with performance indicators (e.g., roundtrip delay, bandwidth, data rates, transfer jitter, affinity and/or anti-affinity), to performance sever 240; Examiner’s Note: Data rate is the measure of how fast data is transferred over a network. In this case, data rate is analogous with migration speed.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar with the teachings of Parker wherein at least one of the first performance metric and the second performance metric comprises a migration speed in order to ensure that performance criteria, in this case migration speed, is being met to ensure efficiency, as discussed in Parker ([0022]).
19. With regard to claim 15, Parker further teaches:
wherein at least one of the first performance metric and the second performance metric comprises a processing resource usage ([0022] Virtual container server 220 may receive instructions from orchestration server 230 to add, remove, combine, or migrate (i.e., move) virtual machines associated with virtual containers stored by virtual container server 220. In some implementations, virtual container server 220 may provide performance data, associated with performance indicators (e.g., roundtrip delay, bandwidth, data rates, transfer jitter, affinity and/or anti-affinity), to performance sever 240; Examiner’s Note Bandwidth is the maximum data transfer rate, which would indicate a processing resource usage. In this case, bandwidth is analogous with processing resource usage.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar with the teachings of Parker wherein at least one of the first performance metric and the second performance metric comprises a processing resource usage in order to ensure that performance criteria, in this case resource usage, is being met to ensure efficiency, as discussed in Parker ([0022]).
20. Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Abali et al. US 20150169337 A1; Wang et al. US 20200042343 A1; Simionescu et al. US 20210089454 A1; and Sampathkumar US 20190188156 A1, as applied in claim 1, in further view of Prinz, III et al. US 20160041996 A1.
21. With regard to claim 16, Abali, Wang, Simionescu, and Sampathkumar teach the apparatus of claim 9 but fail to explicitly teach wherein determining the granularity level comprises selecting a default granularity level.
However, in analogous art, Prinz, III further teaches:
wherein determining the granularity level comprises selecting a default granularity level ([0072] With operator input, the migration plan is determined (430). At least a portion of the migration plan 112 can be based on default parameters (432). In one implementation, for example, the default parameters can select to migrate file system objects with the same structure, organization, and granularity. Additionally, the policies can be replicated from the source filer 20 to the destination filer 50 with the default parameters set to achieve the same granularity (e.g., 1:1 mapping), in terms of logical equivalence, as that provided with the source filer 20; Examiner’s Note: Default parameters, including granularity are chosen for migration.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar with the teachings of Prinz III wherein determining the granularity level comprises selecting a default granularity level in order to migrate file system objects with the same structure, organization, and granularity. This helps the system achieve a 1:1 mapping, as discussed in Prinz III ([0072]).
22. Claims 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Abali et al. US 20150169337 A1; Wang et al. US 20200042343 A1; Simionescu et al. US 20210089454 A1; and Sampathkumar US 20190188156 A1, as applied in claim 1, in further view of Koren et al. US 20230133439 A1.
23. With regard to claim 17, Abali, Wang, Simionescu, and Sampathkumar teach the apparatus of claim 9 but fail to explicitly teach wherein the instructions further cause the processor to: generate a tracking data structure, wherein tracking the portions of the memory of the VM for dirtiness comprises recording, in the tracking data structure, page-level dirtiness of the memory of the VM.
However, in analogous art, Koren teaches:
wherein the instructions further cause the processor to:
generate a tracking data structure, wherein tracking the portions of the memory of the VM for dirtiness comprises recording, in the tracking data structure, page-level dirtiness of the memory of the VM ([0030]; Examiner’s Note: There is a dirty-page data structure that keeps track of dirty pages of the memory of the VM.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar with the teachings of Koren to generate a tracking data structure, wherein tracking the portions of the memory of the VM for dirtiness comprises recording, in the tracking data structure, page-level dirtiness of the memory of the VM in order to keep track of the dirty pages. This allows the system to easily access which pages of the memory are dirty, as discussed in Koren ([0030]).
24. With regard to claim 18, Koren further teaches:
wherein the instructions further cause the processor to:
generate a page-level tracking data structure, wherein tracking the portions of the memory of the VM for dirtiness comprises recording, in the tracking data structure, page identifiers of pages of the memory of the VM which include the dirtied portions ([0035]; [0038]; Examiner’s Note: The addresses are analogous with page identifiers. The addresses of the memory pages are recorded within the dirty-page data structure.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Abali, Wang, Simionescu, and Sampathkumar in view of Koren with the teachings of Koren to generate a page-level tracking data structure, wherein tracking the portions of the memory of the VM for dirtiness comprises recording, in the tracking data structure, page identifiers of pages of the memory of the VM which include the dirtied portions. This enables the system to quickly identify which page is dirtied based on the page identifier, therefore, allowing faster querying, as discussed in Koren ([0035]; [0038]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AN-AN N NGUYEN whose telephone number is (571)272-6147. The examiner can normally be reached Monday-Friday 8:00-5:00 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, AIMEE LI can be reached at (571) 272-4169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AN-AN NGOC NGUYEN/Examiner, Art Unit 2195 /KEVIN L YOUNG/Supervisory Patent Examiner, Art Unit 2194
Prosecution Insights