This patch series adds memory tracking support for performant checkpointing solutions. It can also be used by live migration to improve predictability. Introduction Brendan Cully's Remus project white paper is one of the best written on the subject of fault tolerance using checkpoint/rollback techniques and is the best place to start for a general background. (http://www.cs.ubc.ca/~andy/papers/remus-nsdi-final.pdf) It gives a great outline of the basic requirements and characteristics of a checkpointed system, including a few of the performance issues. But Remus did not go far enough in the area of system performance for commercial production. This patch series addresses known bottleneck and limitation in a checkpointed system: use of large bitmaps to track dirty memory. These bitmaps are copied to userspace when userspace queries KVM for its dirty page information. The use of bitmaps makes sense in the live-migration method, as it is possible for all of memory to be dirtied from one log-dirty pass to another. But in a checkpointed system, the number of dirty pages is bounded such that the VM is paused when it has dirtied a pre-defined number of pages. Traversing a large, sparsely populated bitmap to find set bits is time-consuming, as is copying the bitmap to user-space. The preferred data structure for performant checkpointing solutions is a dense list of guest frame numbers (GFN). This patch series stores the dirty list in kernel memory that can be memory mapped into userspace to allow speedy harvesting. Three new ioctls are implemented to allow userspace to set the dirty log size, to get the dirty count and to rearm the dirty traps. See patch 4 for details. The modification and still more modifications to qemu have allowed us to run checkpoint cycles at rates up to 2500 per second, while still allowing the VM to get useful work done. Design Goals The patch series does not change or remove any existing KVM functionality. It represents only additional functions (ioctls) into KVM from user space and these changes coexist with the current dirty memory logging facilities. It is possible to run multiple guests such that some of the guests perform live migration using the existing memory logging mechanism and others migrate or run in fault tolerant mode using the new memory tracking functions. Modifications All modifications affect only the KVM instance where the primary (active) VM is running, and these modifications are not in play on the standby (passive) host, where a VM is created that matches the primary in its configuration, but it does not execute until a migration/failover event occurs. Patch 1: Introduce new memory tracking ioctls to support performant checkpointing solutions Patch 2: Introduce memory tracking data structures and implement the new ioctls and support functions. Patch 3: Implement dirty list full forcing vcpus to exit. Patch 4: Add documentation for new ioctls. Documentation/virtual/kvm/api.txt | 116 +++++++++++ arch/x86/include/asm/kvm_host.h | 9 + arch/x86/kvm/mmu.c | 7 + arch/x86/kvm/vmx.c | 7 + arch/x86/kvm/x86.c | 13 ++ include/linux/kvm_host.h | 26 +++ include/uapi/linux/kvm.h | 9 + virt/kvm/kvm_main.c | 297 +++++++++++++++++++++++++++- 8 files changed, 480 insertions(+), 4 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
