I recently gathered some performance data when running Windows Server 2008 VMs, and I wanted to share it here. There are 12 Windows Server2008 64-bit VMs (1 vcpu, 2 GB) running which handle the concurrent execution of 6 J2EE type benchmarks. Each benchmark needs a App VM and a Database VM. The benchmark clients inject a fixed rate of requests which yields X% CPU utilization on the host. A different hypervisor was compared; KVM used about 60% more CPU cycles to complete the same amount of work. Both had their hypervisor specific paravirt IO drivers in the VMs. Server is a 2 socket Core/i7, SMT off, with 72 GB memory Host kernel used was kvm.git v2.6.31-rc3-3419-g6df4865 Qemu was kvm-87. I tried a few newer versions of Qemu; none of them worked with the RedHat virtIO Windows drivers. I tried: f3600c589a9ee5ea4c0fec74ed4e06a15b461d52 0.11.0-rc1 0.10.6 kvm-88 All but 0.10.6 had "Problem code 10" driver error in the VM. 0.10.6 had "a disk read error occurred" very early in the booting of the VM. I/O on the host was not what I would call very high: outbound network averaged at 163 Mbit/s inbound was 8 Mbit/s, while disk read ops was 243/sec and write ops was 561/sec Host CPU breakdown was the following: user nice system irq softirq guest idle iowait 5.67 0.00 11.64 0.09 1.05 31.90 46.06 3.59 The amount of kernel time had me concerned. Here is oprofile: > samples % app name symbol name > 1163422 52.3744 kvm-intel.ko vmx_vcpu_run > 103996 4.6816 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 native_set_debugreg > 81036 3.6480 kvm.ko kvm_arch_vcpu_ioctl_run > 37913 1.7068 qemu-system-x86_64 cpu_physical_memory_rw > 34720 1.5630 qemu-system-x86_64 phys_page_find_alloc > 23234 1.0459 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 native_write_msr_safe > 20964 0.9437 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 native_get_debugreg > 17628 0.7936 libc-2.5.so memcpy > 16587 0.7467 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 __down_read > 15681 0.7059 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 __up_read > 15466 0.6962 kvm.ko find_highest_vector > 14611 0.6578 qemu-system-x86_64 qemu_get_ram_ptr > 11254 0.5066 kvm-intel.ko vmcs_writel > 11133 0.5012 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 copy_user_generic_string > 10917 0.4915 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 native_read_msr_safe > 10760 0.4844 qemu-system-x86_64 virtqueue_get_head > 9025 0.4063 kvm-intel.ko vmx_handle_exit > 8953 0.4030 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 schedule > 8753 0.3940 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 fget_light > 8465 0.3811 qemu-system-x86_64 virtqueue_avail_bytes > 8185 0.3685 kvm-intel.ko handle_cr > 8069 0.3632 kvm.ko kvm_set_irq > 7697 0.3465 kvm.ko kvm_lapic_sync_from_vapic > 7586 0.3415 qemu-system-x86_64 main_loop_wait > 7480 0.3367 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 do_select > 7121 0.3206 qemu-system-x86_64 lduw_phys > 7003 0.3153 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 audit_syscall_exit > 6062 0.2729 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 kfree > 5477 0.2466 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 fput > 5454 0.2455 kvm.ko kvm_lapic_get_cr8 > 5096 0.2294 kvm.ko kvm_load_guest_fpu > 5057 0.2277 kvm.ko apic_update_ppr > 4929 0.2219 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 up_read > 4900 0.2206 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 audit_syscall_entry > 4866 0.2191 kvm.ko kvm_apic_has_interrupt > 4670 0.2102 kvm-intel.ko skip_emulated_instruction > 4644 0.2091 kvm.ko kvm_cpu_has_interrupt > 4548 0.2047 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 __switch_to > 4328 0.1948 kvm.ko kvm_apic_accept_pic_intr > 4303 0.1937 libpthread-2.5.so pthread_mutex_lock > 4235 0.1906 vmlinux-2.6.31-rc5-v2.6.31-rc3-3419-g6df4865-autokern1 system_call > 4175 0.1879 kvm.ko kvm_put_guest_fpu > 4170 0.1877 qemu-system-x86_64 ldl_phys > 4098 0.1845 kvm-intel.ko vmx_set_interrupt_shadow > 4003 0.1802 qemu-system-x86_64 kvm_run I was wondering why the get/set debugreg was so high. I don't recall seeing this much with Linux VMs. Here is an average of kvm_stat: > efer_relo 0 > exits 1262814 > fpu_reloa 103842 > halt_exit 9918 > halt_wake 9763 > host_stat 103846 > hypercall 0 > insn_emul 23277 > insn_emul 23277 > invlpg 0 > io_exits 82717 > irq_exits 12797 > irq_injec 18806 > irq_windo 1194 > largepage 12 > mmio_exit 0 > mmu_cache 0 > mmu_flood 0 > mmu_pde_z 0 > mmu_pte_u 0 > mmu_pte_w 0 > mmu_recyc 0 > mmu_shado 0 > mmu_unsyn 0 > nmi_injec 0 > nmi_windo 0 > pf_fixed 12 > pf_guest 0 > remote_tl 0 > request_i 0 > signal_ex 0 > tlb_flush 0 For 12 VMs, do the number of exits/sec seem reasonable? Comments? Thanks, -Andrew -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
