I've been looking at how KVM handles windows guests, and I am a little
concerned with the CPU overhead. My test case is as follows:
I am running 4 instances of a J2EE benchmark. Each instance needs one
application server and one DB server. 8 VMs in total are used.
I have the same App and DB software for Linux and Windows (and same
versions) so I can compare between Linux and Windows. I also have
another hypervisor which I can test both Windows and Linux VMs.
The host has EPT capable processors. VMs in KVM are backed with large
pages.
Test results:
Config CPU utilization
------ -----
KVM-85
Windows Server 2008 64-bit VMs 44.84
RedHat 5.3 w/ 2.6.29 64-bit VMs 24.56
Other-Hypervisor
Windows Server 2008 64-bit VMs 30.63
RedHat 5.3 w/ 2.6.18 64-bit VMs 27.13
-KVM running Windows VMs uses 46% more CPU than the Other-Hypervisor
-The Other-Hypervisor provides an optimized virtual network driver
-KVM results listed above did not use virtio_net or virtio_disk for
Windows, but do for Linux
-One extra KVM run (not listed above) was made with virtio_net for
Windows VMs but only reduced CPU by 2%
-Most of the CPU overhead could be attributed to the DB VMs, where there
is about 5 MB/sec writes per VM
-I don't have a virtio_block driver for Windows to test. Does one exist?
-All tests above had 2 vCPUS per VM
Here's a comparison of kvm_stat between Windows (run1) and Linux (run2):
run1 run2 run1/run2
---- ---- ---------
efer_relo: 0 0 1
exits : 1206880 121916 9.899
fpu_reloa: 210969 20863 10.112
halt_exit: 15092 13222 1.141
halt_wake: 14466 9294 1.556
host_stat: 211066 45117 4.678
hypercall: 0 0 1
insn_emul: 119582 38126 3.136
insn_emul: 0 0 1
invlpg : 0 0 1
io_exits : 131051 26349 4.974
irq_exits: 8128 12937 0.628
irq_injec: 29955 21825 1.373
irq_windo: 2504 2022 1.238
kvm_reque: 0 0 1
largepage: 1 64 0.009
mmio_exit: 59224 0 Inf
mmu_cache: 0 3 0.000
mmu_flood: 0 0 1
mmu_pde_z: 0 0 1
mmu_pte_u: 0 0 1
mmu_pte_w: 0 0 1
mmu_recyc: 0 0 1
mmu_shado: 0 0 1
mmu_unsyn: 0 0 1
mmu_unsyn: 0 0 1
nmi_injec: 0 0 1
nmi_windo: 0 0 1
pf_fixed : 1 67 0.009
pf_guest : 0 0 1
remote_tl: 0 0 1
request_n: 0 0 1
signal_ex: 0 0 1
tlb_flush: 220 14037 0.016
10x the number of exits, a problem?
I happened to try just one vCPU per VM for KVM/Windows VMs, and I was
surprised how much of a difference it made:
Config CPU utilization
------ -----
KVM-85
Windows Server 2008 64-bit VMs, 2 vCPU per VM 44.84
Windows Server 2008 64-bit VMs, 1 vCPU per VM 36.44
A 19% reduction in CPU utilization vs KVM/Windows-2vCPU! Does not
explain all the overhead (vs Other-Hypervisor, 2 vCPUs per VM) but, that
sure seems like a lot between 1 to 2 vCPUs for KVM/Windows-VMs. I have
not run with 1 vCPU per VM with Other-Hypervisor, but I will soon.
Anyway, I also collected kvm_stat for the 1 vCPU case, and here it is
compared to KVM/Linux VMs with 2 vCPUs:
run1 run2 run1/run2
---- ---- ---------
efer_relo: 0 0 1
exits : 1184471 121916 9.715
fpu_reloa: 192766 20863 9.240
halt_exit: 4697 13222 0.355
halt_wake: 4360 9294 0.469
host_stat: 192828 45117 4.274
hypercall: 0 0 1
insn_emul: 130487 38126 3.422
insn_emul: 0 0 1
invlpg : 0 0 1
io_exits : 114430 26349 4.343
irq_exits: 7075 12937 0.547
irq_injec: 29930 21825 1.371
irq_windo: 2391 2022 1.182
kvm_reque: 0 0 1
largepage: 0 64 0.001
mmio_exit: 69028 0 Inf
mmu_cache: 0 3 0.000
mmu_flood: 0 0 1
mmu_pde_z: 0 0 1
mmu_pte_u: 0 0 1
mmu_pte_w: 0 0 1
mmu_recyc: 0 0 1
mmu_shado: 0 0 1
mmu_unsyn: 0 0 1
mmu_unsyn: 0 0 1
nmi_injec: 0 0 1
nmi_windo: 0 0 1
pf_fixed : 0 67 0.001
pf_guest : 0 0 1
remote_tl: 0 0 1
request_n: 0 0 1
signal_ex: 0 0 1
tlb_flush: 124 14037 0.009
Still see the huge difference in vm_exits, so I guess not all is great yet.
So, what would be the best course of action for this? Is there a
virtio_block driver to test? Can we find the root cause of the exits
(is there a way to get stack dump or something that can show where there
are coming from)?
Thanks!
-Andrew
P.S. Here is the qemu cmd line for the windows VMs:
/usr/local/bin/qemu-system-x86_64 -name newcastle-xdbt01 -hda
/dev/disk/by-id/scsi-3600a0b80000f1eb10000074f4a02b08a -net
nic,model=e1000,vlan=0,macaddr=00:50:56:00:00:03 -m 1024 -mempath
/hugetlbfs -net tap,vlan=0,ifname=tap3,script=/etc/qemu-ifup -vnc
127.0.0.1:3 -smp 2 -daemonize
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
