On 04/20/11 20:35, Alex Williamson wrote:
> Device assignment via a VF provides the lowest latency and most
> bandwidth for *getting data off the host system*, though virtio/vhost is
> getting better. If all you care about is VM-VM on the same host or
> VM-host, then virtio is only limited by memory bandwidth/latency and
> host processor cycles. Your processor has 25GB/s of memory bandwidth.
> On the other hand, the VF has to send data all the way out to the wire
> and all the way back up through the NIC to get to the other VM/host.
> You're using a 1Gb/s NIC. Your results actually seem to indicate you're
> getting better than wire rate, so maybe you're only passing through an
> internal switch on the NIC, in any case, VFs are not optimal for
> communication within the same physical system. They are optimal for off
> host communication. Thanks,
Hi Alex:
Host-host was the next focus for the tests. I have 2 of the
aforementioned servers, each configured identically. As a reminder:
Host:
Dell R410
2 quad core E5620@xxxx GHz processors
16 GB RAM
Intel 82576 NIC (Gigabit ET Quad Port)
- devices eth2, eth3, eth4, eth5
Fedora 14
kernel: 2.6.35.12-88.fc14.x86_64
qemu-kvm.git, ffce28fe6 (18-April-11)
VMs:
Fedora 14
kernel 2.6.35.11-83.fc14.x86_64
2 vcpus
1GB RAM
2 NICs - 1 virtio, 1 VF
The virtio network arguments to qemu-kvm are:
-netdev type=tap,vhost=on,ifname=tap0,id=netdev1
-device virtio-net-pci,mac=${mac},netdev=netdev1
For this round of tests I have the following setup:
.======================================.
| Host - A |
| |
| .-------------------------. |
| | Virtual Machine - C | |
| | | |
| | .------. .------. | |
| '--| eth1 |-----| eth0 |--' |
| '------' '------' |
| 192.168. | | 192.168.103.71
| 102.71 | .------. |
| | | tap0 | |
| | '------' |
| | | |
| | .------. |
| | | br | 192.168.103.79
| | '------' |
| {VF} | |
| .--------. .------. |
'=======| eth2 |======| eth3 |======='
'--------' '------'
192.168.102.79 | |
| point-to- |
| point |
| connections |
192.168.102.80 | |
.--------. .------.
.=======| eth2 |======| eth3 |=======.
| '--------' '------' |
| {VF} | |
| | .------. |
| | | br | 192.168.103.80
| | '------' |
| | | |
| | .------. |
| | | tap0 | |
| 192.168. | '------' |
| 102.81 | | 192.168.103.81
| .------. .------. |
| .--| eth1 |-----| eth0 |--. |
| | '------' '------' | |
| | | |
| | Virtual Machine - D | |
| '-------------------------' |
| |
| Host - B |
'======================================'
So, basically, 192.168.102 is the network where the VMs have a VF, and
192.168.103 is the network where the VMs use virtio for networking.
The netperf commands are all run on either Host-A or VM-C:
netperf -H $ip -jcC -v 2 -t TCP_RR -- -r 1024 -D L,R
netperf -H $ip -jcC -v 2 -t TCP_STREAM -- -m 1024 -D L,R
latency throughput
(usec) Mbps
cross-host:
A-B, eth2 185 932
A-B, eth3 185 935
same host, host-VM:
A-C, using VF 488 1085 (seen as high as 1280's)
A-C, virtio 150 4282
cross-host, host-VM:
A-D, VF 489 938
A-D, virtio 288 889
cross-host, VM-VM:
C-D, VF 488 934
C-D, virtio 490 933
While throughput for VFs is fine (near line-rate when crossing hosts),
the latency is horrible. Any options to improve that?
David
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
