On 13/09/2010 19:34, Alex Williamson wrote: > On Mon, Sep 13, 2010 at 4:32 AM, Thibault VINCENT > <thibault.vincent@xxxxxxxxxxxx> wrote: >> Hello >> >> I'm trying to achieve higher than gigabit transferts over a virtio NIC >> with no success, and I can't find a recent bug or discussion about such >> an issue. >> >> The simpler test consist of two VM running on a high-end blade server >> with 4 cores and 4GB RAM each, and a virtio NIC dedicated to the >> inter-VM communication. On the host, the two vnet interfaces are >> enslaved into a bridge. I use a combination of 2.6.35 on the host and >> 2.6.32 in the VMs. >> Running iperf or netperf on these VMs, with TCP or UDP, result in >> ~900Mbits/s transferts. This is what could be expected of a 1G >> interface, and indeed the e1000 emulation performs similar. >> >> Changing the txqueuelen, MTU, and offloading settings on every interface >> (bridge/tap/virtio_net) didn't improve the speed, nor did the >> installation of irqbalance and the increase in CPU and RAM. >> >> Is this normal ? Is the multiple queue patch intended to address this ? >> It's quite possible I missed something :) > > I'm able to achieve quite a bit more than 1Gbps using virtio-net > between 2 guests on the same host connected via an internal bridge. > With the virtio-net TX bottom half handler I can easily hit 7Gbps TCP > and 10+Gbps UDP using netperf (TCP_STREAM/UDP_STREAM tests). Even > without the bottom half patches (not yet in qemu-kvm.git), I can get > ~5Gbps. Maybe you could describe your setup further, host details, > bridge setup, guests, specific tests, etc... Thanks, Thanks Alex, I don't use the bottom half patches but anything between 3Gbps and 5Gbps would be fine. Here are some more details: Host ----- Dell M610 ; 2 x Xeon X5650 ; 6 x 8GB Debian Squeeze amd64 qemu-kvm 0.12.5+dfsg-1 kernel 2.6.35-1 amd64 (Debian Experimental) Guests ------- Debian Squeeze amd64 kernel 2.6.35-1 amd64 (Debian Experimental) To measure the throughput between the guests, I do the following. On the host: * create a bridge # brctl addbr br_test # ifconfig br_test 1.1.1.1 up * start two guests # kvm -enable-kvm -m 4096 -smp 4 -drive file=/dev/vg/deb0,id=0,boot=on,format=raw -device virtio-blk-pci,drive=0,id=0 -device virtio-net-pci,vlan=0,id=1,mac=52:54:00:cf:6a:b0 -net tap,vlan=0,name=hostnet0 # kvm -enable-kvm -m 4096 -smp 4 -drive file=/dev/vg/deb1,id=0,boot=on,format=raw -device virtio-blk-pci,drive=0,id=0 -device virtio-net-pci,vlan=0,id=1,mac=52:54:00:cf:6a:b1 -net tap,vlan=0,name=hostnet0 * add guests to the bridge # brctl addif br_test tap0 # brctl addif br_test tap1 On the first guest: # ifconfig eth0 1.1.1.2 up # iperf -s -i 1 On the second guest: # ifconfig eth0 1.1.1.3 up # iperf -c 1.1.1.2 -i 1 ------------------------------------------------------------ Client connecting to 1.1.1.2, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.3 port 43510 connected with 1.1.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 80.7 MBytes 677 Mbits/sec [ 3] 1.0- 2.0 sec 102 MBytes 855 Mbits/sec [ 3] 2.0- 3.0 sec 101 MBytes 847 Mbits/sec [ 3] 3.0- 4.0 sec 104 MBytes 873 Mbits/sec [ 3] 4.0- 5.0 sec 104 MBytes 874 Mbits/sec [ 3] 5.0- 6.0 sec 105 MBytes 881 Mbits/sec [ 3] 6.0- 7.0 sec 103 MBytes 862 Mbits/sec [ 3] 7.0- 8.0 sec 101 MBytes 848 Mbits/sec [ 3] 8.0- 9.0 sec 105 MBytes 878 Mbits/sec [ 3] 9.0-10.0 sec 105 MBytes 882 Mbits/sec [ 3] 0.0-10.0 sec 1011 MBytes 848 Mbits/sec On the host again: # iperf -c 1.1.1.1 -i 1 ------------------------------------------------------------ Client connecting to 1.1.1.1, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.3 port 60456 connected with 1.1.1.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 97.9 MBytes 821 Mbits/sec [ 3] 1.0- 2.0 sec 136 MBytes 1.14 Gbits/sec [ 3] 2.0- 3.0 sec 153 MBytes 1.28 Gbits/sec [ 3] 3.0- 4.0 sec 160 MBytes 1.34 Gbits/sec [ 3] 4.0- 5.0 sec 156 MBytes 1.31 Gbits/sec [ 3] 5.0- 6.0 sec 122 MBytes 1.02 Gbits/sec [ 3] 6.0- 7.0 sec 121 MBytes 1.02 Gbits/sec [ 3] 7.0- 8.0 sec 137 MBytes 1.15 Gbits/sec [ 3] 8.0- 9.0 sec 139 MBytes 1.17 Gbits/sec [ 3] 9.0-10.0 sec 140 MBytes 1.17 Gbits/sec [ 3] 0.0-10.0 sec 1.33 GBytes 1.14 Gbits/sec You can see it's quite slow compared to your figures, between the guests and with the host too. And there is no specific load on any of the three systems, htop in a guest only report one of the four cores going up to 70% (sys+user+wait) during the test. The other tests I mentioned are: * iperf or netperf over UDP : maybe 10% faster, no more * interface settings : very very few effect # ifconfig [br_test,tap0,tap1,eth0] txqueuelen 20000 # ifconfig eth0 mtu 65534 <-- guest only # ethtool -K eth0 gso on <-- guest only * double the RAM or number of CPU in the guests, no effect * run the two guests on separate hosts, linked with a 10G net, again it's exactly the same throughput. I can get >7Gbps between the hosts. Then I reproduced the test on a completely different system, a desktop with Intel i5, 4GB, Debian Squeeze. And unfortunately I get the same figures, so the limitation doesn't seem to be hardware bounded. What's your distro, kernel, and kvm version Alex ? Do you think I need to compile qemu with a specific patch or option that may be missing in the Squeeze source ? Thanks! -- Thibault VINCENT SmartJog S.A.S. - Groupe TDF - Pôle multimédia 27 Bd Hippolyte Marques, 94200 Ivry sur Seine, France Phone : +33.1.58.68.62.38 Fax : +33.1.58.68.60.97
Attachment:
signature.asc
Description: OpenPGP digital signature
