On Fri, Dec 14, 2018 at 06:24:40PM +0800, jiangyiwen wrote: > On 2018/12/12 23:09, Michael S. Tsirkin wrote: > > On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen wrote: > >> Now vsock only support send/receive small packet, it can't achieve > >> high performance. As previous discussed with Jason Wang, I revisit the > >> idea of vhost-net about mergeable rx buffer and implement the mergeable > >> rx buffer in vhost-vsock, it can allow big packet to be scattered in > >> into different buffers and improve performance obviously. > >> > >> This series of patches mainly did three things： > >> - mergeable buffer implementation > >> - increase the max send pkt size > >> - add used and signal guest in a batch > >> > >> And I write a tool to test the vhost-vsock performance, mainly send big > >> packet(64K) included guest->Host and Host->Guest. I test performance > >> independently and the result as follows: > >> > >> Before performance: > >> Single socket Multiple sockets(Max Bandwidth) > >> Guest->Host ~400MB/s ~480MB/s > >> Host->Guest ~1450MB/s ~1600MB/s > >> > >> After performance only use implement mergeable rx buffer: > >> Single socket Multiple sockets(Max Bandwidth) > >> Guest->Host ~400MB/s ~480MB/s > >> Host->Guest ~1280MB/s ~1350MB/s > >> > >> In this case, max send pkt size is still limited to 4K, so Host->Guest > >> performance will worse than before. > > > > It's concerning though, what if application sends small packets? > > What is the source of the slowdown? Do you know? > > > > Hi Michael, > > To the two cases, I test the results included small and big packets as > follows: > > 64K packets performance comparison: > Single socket Multiple sockets > Host->Guest(before) 1352.60MB/s 1436.33MB/s > > > Host->Guest(only use mergeable rx buffer) 1290.08MB/s 1212.67MB/s > > 4K packets performance comparison: > Single socket Multiple sockets > Host->Guest(before) 535.47MB/s 688.67MB/s > Host->Guest(only use mergeable rx buffer) 522.33MB/s 599.00MB/s > > 3K packets performance comparison: > Single socket Multiple sockets > Host->Guest(before) 359.74MB/s 442.00MB/s > Host->Guest(only use mergeable rx buffer) 374.47MB/s 452.33MB/s > > We can see an interesting thing, for 64K and 4K packets, > using mergeable buffer has a poor performance, for 3K packet, > both have the same performance. > > I guess in mergeable mode, when host send a 4k packet to guest, we > should call vhost_get_vq_desc() twice in host(hdr + 4k data), > and in guest we also should call virtqueue_get_buf() twice. So > when packet is smaller than (4k - hdr), it can be packed in a > single page, so the performance is the same as before. > > So in the mergeable mode, the performance may be > worse in ((4k - hdr), 4k] than before. > > Thanks, > Yiwen. The conclusion seems to be that mergeable buffers themselves only hurt performance, but they allow batching which improves performance. So let's add batching without mergeable buffers then? > >> After performance increase the max send pkt size to 64K: > >> Single socket Multiple sockets(Max Bandwidth) > >> Guest->Host ~1700MB/s ~2900MB/s > >> Host->Guest ~1500MB/s ~2440MB/s > >> > >> After performance all patches are used: > >> Single socket Multiple sockets(Max Bandwidth) > >> Guest->Host ~1700MB/s ~2900MB/s > >> Host->Guest ~1700MB/s ~2900MB/s > >> > >> >From the test results, the performance is improved obviously, and guest > >> memory will not be wasted. > >> > >> In addition, in order to support mergeable rx buffer in virtio-vsock, > >> we need to add a qemu patch to support parse feature. > >> > >> --- > >> v1 -> v2: > >> * Addressed comments from Jason Wang. > >> * Add performance test result independently. > >> * Use Skb_page_frag_refill() which can use high order page and reduce > >> the stress of page allocator. > >> * Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small > >> size can't fill one full packet, we only 128 vq num now. > >> * Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx > >> consistency. > >> * Add virtio_transport ops to get max pkt len, in order to be compatible > >> with old version. > >> --- > >> > >> Yiwen Jiang (5): > >> VSOCK: support fill mergeable rx buffer in guest > >> VSOCK: support fill data to mergeable rx buffer in host > >> VSOCK: support receive mergeable rx buffer in guest > >> VSOCK: increase send pkt len in mergeable mode to improve performance > >> VSOCK: batch sending rx buffer to increase bandwidth > >> > >> drivers/vhost/vsock.c | 183 ++++++++++++++++++++----- > >> include/linux/virtio_vsock.h | 13 +- > >> include/uapi/linux/virtio_vsock.h | 5 + > >> net/vmw_vsock/virtio_transport.c | 229 +++++++++++++++++++++++++++----- > >> net/vmw_vsock/virtio_transport_common.c | 66 ++++++--- > >> 5 files changed, 411 insertions(+), 85 deletions(-) > >> > >> -- > >> 1.8.3.1 > > > > . > > >
