Hello KVM developers, I have a question about the dirty log mechanism in qemu. I am doing some research on multi-core VM fault tolerance and I extended qemu to do checkpoint-restart based replication (something similar to the Remus project on Xen). I am using a great amount of the live migration code, particularly the dirty page tracking mechanism. However, when I run a multi-core (up to 16 cores) VM in dirty log mode I see significant slowdown of the dirty page tracking. I identified that the problem is in the cpu_physical_memory_reset_dirty() function which iterates all vcpu's soft-mmu TLB entries, essentially for every single address that was reported as dirty by the kernel. My understanding is that dirty page tracking is carried out entirely by the KVM kernel module in case two dimensional page tables are supported by the hardware (I have NPT enabled). This log is then synchronized with userspace in the cpu_physical_sync_dirty_bitmap() function which updates the ram_list.phys_dirty buffer in qemu. Could someone explain/confirm why is it necessary to clear the soft-mmu TLBs in cpu_physical_memory_reset_dirty() ? Are those being used in case of tdp? I suspect it's unnecessary if the hardware supports MMU virtualization. I modified this code to clear only ram_list.phys_dirty and ran a couple of tests on live migrating multi-core VMs in such way, they seem to work fine and I get much better dirty-log performance. (This is important when doing replication because the VM has to be executed in log-dirty mode at all times.) Please let me know if I am missing something! I run vanilla kernel 2.6.37 with the latest qemu version from the git repository (version 0.14.50 (qemu-kvm-devel)). My hardware config is: processor : 15 vendor_id : AuthenticAMD cpu family : 16 model : 4 model name : Quad-Core AMD Opteron(tm) Processor 8378 stepping : 2 cpu MHz : 2400.401 cache size : 512 KB physical id : 3 siblings : 4 core id : 3 cpu cores : 4 apicid : 15 initial apicid : 15 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save bogomips : 4800.25 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate and I have NPT support enabled in the KVM kernel module. Thanks, Balazs -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
