I have a working copy of libvirt 0.10.2 + qemu 1.2 installed on a vanilla
up-to-date (2.6.32-279.9.1) CentOS 6 host, and get very good VM <-> VM
network performance (both running on the same host) using virtio. I have
cgroups set to cap the VMs at 10Gbps and iperf shows I'm getting exactly
10Gbps.
I copied these VMs to a CentOS 5 host and installed libvirt 1.0 + qemu 1.2.
However, the best performance I can get in between the VMs (again running on
the same host) is ~2Gbps. In both cases, this is over a bridged interface
with static IPs assigned to each VM. I've also tried virtual networking
with NAT or routing, yielding the same results.
I figured it was due to vhost-net missing on the older CentOS 5 kernel, so I
installed 2.6.39-4.2 from ELRepo and got the /dev/vhost-net device and vhost
processes associated with each VM:
]$ lsmod | grep vhost
vhost_net 28446 2
tun 23888 7 vhost_net
]$ ps aux | grep vhost-
root 9628 0.0 0.0 0 0 ? S 17:57 0:00
[vhost-9626]
root 9671 0.0 0.0 0 0 ? S 17:57 0:00
[vhost-9670]
]$ ls /dev/vhost-net -al
crw------- 1 root root 10, 58 Nov 13 15:19 /dev/vhost-net
After installing the new kernel, I also tried rebuilding libvirt and qemu,
to no avail. I also disabled cgroups, just in case it was getting in the
way, as well as iptables. I can see the virtio_net module loaded inside the
guest, and using virtio raises my performance from <400Mbps to 2Gbps, so it
does make some improvement.
The only differences between the two physical hosts that I can find are:
- qemu on the CentOS 5 host builds without preadv support - would this make
such a huge performance difference? CentOS5 only comes with an old version
of glibc, which is missing preadv
- qemu on the CentOS 5 host builds without PIE
- libvirt 1.0 was required on the CentOS 5 host, since 0.10.2 had a build
bug. This shouldn't matter I don't think.
- I haven't tried rebuilding the VMs from scratch on the CentOS5 host, which
I guess is worth a try.
The qemu process is being started with virtio + vhost:
/usr/bin/qemu-system-x86_64 -name vmname -S -M pc-1.2 -enable-kvm -m 4096
-smp 8,sockets=8,cores=1,threads=1 -uuid
212915ed-a34a-4d6d-68f5-2216083a7693 -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vmname.monitor,server,nowai
t -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/mnt/vmname/disk.img,if=none,id=drive-virtio-disk0,format=raw,cache=non
e -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virti
o-disk0,bootindex=1 -netdev tap,fd=16,id=hostnet0,vhost=on,vhostfd=18
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:11:22:33:44:55,bus=pci.0,addr=
0x3 -chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc
127.0.0.1:1 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
The relevant part of my libvirt config, of which I've tried omitting the
target, alias and address elements with no difference in performance:
<interface type="bridge">
<mac address="00:11:22:33:44:55"/>
<source bridge="br0"/>
<target dev="vnet0"/>
<model type="virtio"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03"
function="0x0"/>
</interface>
Is there something else which could be getting in the way here?
Thanks!
Ben Clay
rbclay@xxxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
