Following set of patches implement transmit MQ in virtio-net. Also
included is the user qemu changes. MQ is disabled by default unless
qemu specifies it.
Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in
virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line
aligned sq[0] to sq[n]. Another option was to define
'send_queue' as:
struct send_queue {
struct virtqueue *svq;
struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
} ____cacheline_aligned_in_smp;
and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope
the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX]
handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX
for single stream case.
Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an
optional 'numtxqs' option. e.g. for a smp=4 guest:
vhost=on -> #txqueues = 1
vhost=on,mq=on -> #txqueues = 4
vhost=on,mq=on,numtxqs=2 -> #txqueues = 2
vhost=on,mq=on,numtxqs=8 -> #txqueues = 8
Performance (guest -> local host):
-----------------------------------
System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except
when number of netperf sessions is 1, which has 10 runs of 12 secs
each). No tuning (default netperf) other than taskset vhost's to
cpus 0-3. numtxqs=32 gave the best results though the guest had
only 4 vcpus (I haven't tried beyond that).
______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 4.46 -1.96 .19 -12.50 -6.06
2 4.93 -1.16 2.10 0 -2.38
4 46.17 64.77 33.72 19.51 -2.48
8 47.89 70.00 36.23 41.46 13.35
16 48.97 80.44 40.67 21.11 -5.46
24 49.03 78.78 41.22 20.51 -4.78
32 51.11 77.15 42.42 15.81 -6.87
40 51.60 71.65 42.43 9.75 -8.94
48 50.10 69.55 42.85 11.80 -5.81
64 46.24 68.42 42.67 14.18 -3.28
80 46.37 63.13 41.62 7.43 -6.73
96 46.40 63.31 42.20 9.36 -4.78
128 50.43 62.79 42.16 13.11 -1.23
________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%
______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 -.76 -1.56 2.33 0 3.03
2 17.41 11.11 11.41 0 -4.76
4 42.12 55.11 30.20 19.51 .62
8 54.69 80.00 39.22 24.39 -3.88
16 54.77 81.62 40.89 20.34 -6.58
24 54.66 79.68 41.57 15.49 -8.99
32 54.92 76.82 41.79 17.59 -5.70
40 51.79 68.56 40.53 15.31 -3.87
48 51.72 66.40 40.84 9.72 -7.13
64 51.11 63.94 41.10 5.93 -8.82
80 46.51 59.50 39.80 9.33 -4.18
96 47.72 57.75 39.84 4.20 -7.62
128 54.35 58.95 40.66 3.24 -8.63
________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%
______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 -1.43 -3.52 1.55 0 3.03
2 33.09 21.63 20.12 -10.00 -9.52
4 67.17 94.60 44.28 19.51 -11.80
8 75.72 108.14 49.15 25.00 -10.71
16 80.34 101.77 52.94 25.93 -4.49
24 70.84 93.12 43.62 27.63 -5.03
32 69.01 94.16 47.33 29.68 -1.51
40 58.56 63.47 25.91 -3.92 -25.85
48 61.16 74.70 34.88 .89 -22.08
64 54.37 69.09 26.80 -6.68 -30.04
80 36.22 22.73 -2.97 -8.25 -27.23
96 41.51 50.59 13.24 9.84 -16.77
128 48.98 38.15 6.41 -.33 -22.80
________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%
______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 7.62 -38.03 -26.26 -50.00 -33.33
2 28.95 20.46 21.62 0 -7.14
4 84.05 60.79 45.74 -2.43 -12.42
8 86.43 79.57 50.32 15.85 -3.10
16 88.63 99.48 58.17 9.47 -13.10
24 74.65 80.87 41.99 -1.81 -22.89
32 63.86 59.21 23.58 -18.13 -36.37
40 64.79 60.53 22.23 -15.77 -35.84
48 49.68 26.93 .51 -36.40 -49.61
64 54.69 36.50 5.41 -26.59 -43.23
80 45.06 12.72 -13.25 -37.79 -52.08
96 40.21 -3.16 -24.53 -39.92 -52.97
128 36.33 -33.19 -43.66 -5.68 -20.49
________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%
Signed-off-by: Krishna Kumar <krkumar2@xxxxxxxxxx>
---
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
