On Tue, Apr 16, 2013 at 04:30:17PM -0400, Mason Turner wrote: > We have an in-house app, written in c, that is not performing as well as we'd hoped it would when moving to a VM. We've tried all the common tuning recommendations (virtio, tap interface, cpu pining), without any change in performance. Even terminating all of the other VMs on the host doesn't make a difference. The VM doesn't appear to be CPU, memory or IO bound. We are trying to maximize UDP-based QPS against the in-house app. > > I've been running strace against the app and "perf kvm" against the VM to try to identify any bottlenecks. I would say there are a lot of kvm_exits, but I'm not sure how to quantify what is acceptable and what is not. > > We are trying to maximize UDP queries against the app. I've read a few times that the virtio network stack results in a lot of vm_exits. Unfortunately, we can't use the direct PCI access with our hardware. Can you explain the traffic characteristics more? * UDP packet size * Pattern: 1 query packet, 1 response packet or something more exotic * Bare metal QPS (the goal) * Guest QPS (what you're seeing) * Benchmark configuration: are packets going across a physical network? > Is there a good resource "inefficient" system calls? Things that result in higher than normal kvm_exits, or other performance killers? > > Thanks for the help. > > Our hypdervisor is running on > CentOS 6.3: 2.6.32-279.22.1.el6.x86_64 > qemu-kvm 0.12.1.2 > libvirt 0.9.10 > > Our app is running on > Centos 6.1: 2.6.32-131.0.15.el6.x86_64 Slightly outdated guest and host. It might be worth trying upstream kernels and QEMU (build from source). > <domain type='kvm'> > <name>thing1</name> > <uuid>abe76ce9-60a0-4727-a7ae-cf572e5c3f21</uuid> > <memory unit='KiB'>16384000</memory> > <currentMemory unit='KiB'>16384000</currentMemory> > <vcpu placement='static'>6</vcpu> > <cputune> > <vcpupin vcpu='0' cpuset='0'/> > <vcpupin vcpu='1' cpuset='2'/> > <vcpupin vcpu='2' cpuset='4'/> > <vcpupin vcpu='3' cpuset='6'/> > <vcpupin vcpu='4' cpuset='8'/> > <vcpupin vcpu='5' cpuset='10'/> > </cputune> > <numatune> > <memory mode='interleave' nodeset='0,2,4,6,8,10'/> > </numatune> > <os> > <type arch='x86_64' machine='rhel6.0.0'>hvm</type> > <boot dev='hd'/> > </os> > <features> > <acpi/> > <apic/> > <pae/> > </features> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <devices> > <emulator>/usr/libexec/qemu-kvm</emulator> > <disk type='file' device='disk'> > <driver name='qemu' type='raw' cache='none'/> > <source file='/var/lib/libvirt/images/thing1-disk0'/> > <target dev='vda' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </disk> > <controller type='usb' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> > </controller> > <interface type='bridge'> > <mac address='00:5e:e3:e1:8a:aa'/> > <source bridge='virbr0'/> > <model type='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> > </interface> Please double-check that vhost-net is being used: http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbpvhostnet.htm > <serial type='pty'> > <target port='0'/> > </serial> > <console type='pty'> > <target type='serial' port='0'/> > </console> > <input type='tablet' bus='usb'/> > <input type='mouse' bus='ps2'/> > <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/> > <video> > <model type='cirrus' vram='9216' heads='1'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> > </video> > <memballoon model='virtio'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> > </memballoon> > </devices> > </domain> -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
