This patch series adds ring-based dirty memory tracking support for performant checkpointing solutions. It can also be used by live migration to improve predictability. Introduction Brendan Cully's Remus project white paper is one of the best written on the subject of fault tolerance using checkpoint/rollback techniques and is the best place to start for a general background. (http://www.cs.ubc.ca/~andy/papers/remus-nsdi-final.pdf) It gives a great outline of the basic requirements and characteristics of a checkpointed system, including a few of the performance issues. But Remus did not go far enough in the area of system performance for commercial production. This patch series addresses known bottleneck and limitation in a checkpointed system: use of large bitmaps to track dirty memory. These bitmaps are copied to userspace when userspace queries KVM for its dirty page information. The use of bitmaps makes sense in the live-migration method, as it is possible for all of memory to be dirtied from one log-dirty pass to another. But in a checkpointed system, the number of dirty pages is bounded such that the VM is paused when it has dirtied a pre-defined number of pages. Traversing a large, sparsely populated bitmap to find set bits is time-consuming, as is copying the bitmap to user-space. The preferred data structure for performant checkpointing solutions is a dense list of guest frame numbers (GFN). This patch series stores the dirty list in kernel memory that can be memory mapped into userspace to allow speedy harvesting. The modification and still more modifications to qemu have allowed us to run checkpoint cycles at rates up to 2500 per second, while still allowing the VM to get useful work done. Design Goals The patch series does not change or remove any existing KVM functionality. It represents only additional functions (ioctls) into KVM from user space and these changes coexist with the current dirty memory logging facilities. It is possible to run multiple guests such that some of the guests perform live migration using the existing memory logging mechanism and others migrate or run in fault tolerant mode using the new memory tracking functions. Modifications All modifications affect only the KVM instance where the primary (active) VM is running, and these modifications are not in play on the standby (passive) host, where a VM is created that matches the primary in its configuration, but it does not execute until a migration/failover event occurs. Patch 1: Add support for capabilities that can be enabled in a generic way. Instroduce new capability: ring-based dirty memory logging Patch 2: Add new data type, struct kvm_gfn_ring, and support functions for ring-based dirty memory logging. Add new ioctl, KVM_RESET_DIRTY_PAGES, for dirty trap reset. Patch 3: Modify kvm_write_guest_cached() and kvm_write_guest_offset_cached() to take vcpu as a parameter instead kvm. Patch 4: Add new exit reason KVM_EXIT_DIRTY_LOG_FULL for dirty ring full conditions. Patch 5: Implement ring-base dirty memory tracking. Documentation/virtual/kvm/api.txt | 94 +++++++++- arch/powerpc/kvm/powerpc.c | 14 +- arch/s390/kvm/kvm-s390.c | 11 +- arch/x86/include/asm/kvm_host.h | 5 + arch/x86/kvm/Makefile | 3 +- arch/x86/kvm/lapic.c | 4 +- arch/x86/kvm/mmu.c | 7 + arch/x86/kvm/vmx.c | 7 + arch/x86/kvm/x86.c | 36 ++-- include/linux/kvm_gfn_ring.h | 37 ++++ include/linux/kvm_host.h | 20 ++- include/uapi/linux/kvm.h | 33 ++++ virt/kvm/gfn_ring.c | 100 +++++++++++ virt/kvm/kvm_main.c | 267 ++++++++++++++++++++++++++-- 14 files changed, 569 insertions(+), 69 deletions(-)
