Dear List, I'm a PhD student researching about virtualization. I'm currently working on performance degradation that is caused by neighboring VMs. I am running KVM 1.7.1 on Debian with 3.10.0-2-pve kernel (proxmox). Hardware is a core i7 server with 8GB of RAM. KVM runs 4 ubuntu 12.04 VMs. Each has 4 vcpu and 4GB of RAM. I run various Parsec Benchmarks on them to see the effect of neighboring VMs on the run time of each other. For "dedup" benchmark (a parallel, pipelined program; de-duplication algorithm), I'm seeing a huge performance degradation which I didn't understand. For example, - when "canneal", "raytrace" and "blackscholes" runs in neighboring VMs, the run time of dedup is 15.13sec. - when "streamcluster", "vips" and "ferret" runs in neighboring VMs, the run time of dedup is 166sec. ! My question is why is this happening!? I've analyzed the situation with perf tool to investigate hardware/kernel/kvm performance counters and I found that surprisingly the number of (total) executed instructions varies from 62,265,887,667 to 250,020,356,568 for the above example. I also found that a very large portion of the above difference comes from instructions that run in kernel mode (:u and :k of perf events). I am guessing this is because of locks that dedup spins on them. I've learned that there is "Pause Loop Exiting" feature in KVM (am I right?) so I guess vcpu that is spinning should be swapped out. But I guess this is not happening... Am I mistaking? I've pinned (using taskset -a) dedup's VM on 4 CPUs and the other 3 VMs on the other 4 CPUs: The huge difference between runtimes gone. So, I guess this is because of scheduling (and not LLC contention, for e.g.). I've also learned that KVM uses Linux's CFS... However, I don't understand why scheduler's behavior differs with neighboring VMs? I really appreciate any help :) Best Seyed Hossein -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
