On Fri, Sep 29, 2017 at 9:25 PM, Willem de Bruijn <willemdebruijn.kernel@xxxxxxxxx> wrote: > On Fri, Sep 29, 2017 at 3:38 PM, Michael S. Tsirkin <mst@xxxxxxxxxx> wrote: >> On Wed, Sep 27, 2017 at 08:25:56PM -0400, Willem de Bruijn wrote: >>> From: Willem de Bruijn <willemb@xxxxxxxxxx> >>> >>> Vhost-net has a hard limit on the number of zerocopy skbs in flight. >>> When reached, transmission stalls. Stalls cause latency, as well as >>> head-of-line blocking of other flows that do not use zerocopy. >>> >>> Instead of stalling, revert to copy-based transmission. >>> >>> Tested by sending two udp flows from guest to host, one with payload >>> of VHOST_GOODCOPY_LEN, the other too small for zerocopy (1B). The >>> large flow is redirected to a netem instance with 1MBps rate limit >>> and deep 1000 entry queue. >>> >>> modprobe ifb >>> ip link set dev ifb0 up >>> tc qdisc add dev ifb0 root netem limit 1000 rate 1MBit >>> >>> tc qdisc add dev tap0 ingress >>> tc filter add dev tap0 parent ffff: protocol ip \ >>> u32 match ip dport 8000 0xffff \ >>> action mirred egress redirect dev ifb0 >>> >>> Before the delay, both flows process around 80K pps. With the delay, >>> before this patch, both process around 400. After this patch, the >>> large flow is still rate limited, while the small reverts to its >>> original rate. See also discussion in the first link, below. >>> >>> The limit in vhost_exceeds_maxpend must be carefully chosen. When >>> vq->num >> 1, the flows remain correlated. This value happens to >>> correspond to VHOST_MAX_PENDING for vq->num == 256. Allow smaller >>> fractions and ensure correctness also for much smaller values of >>> vq->num, by testing the min() of both explicitly. See also the >>> discussion in the second link below. >>> >>> Link:http://lkml.kernel.org/r/CAF=yD-+Wk9sc9dXMUq1+x_hh=3ThTXa6BnZkygP3tgVpjbp93g@xxxxxxxxxxxxxx >>> Link:http://lkml.kernel.org/r/20170819064129.27272-1-den@xxxxxxxxxxxxx >>> Signed-off-by: Willem de Bruijn <willemb@xxxxxxxxxx> >> >> I'd like to see the effect on the non rate limited case though. >> If guest is quick won't we have lots of copies then? > > Yes, but not significantly more than without this patch. > > I ran 1, 10 and 100 flow tcp_stream throughput tests from a sender > in the guest to a receiver in the host. > > To answer the other benchmark question first, I did not see anything > noteworthy when increasing vq->num from 256 to 1024. > > With 1 and 10 flows without this patch all packets use zerocopy. > With the patch, less than 1% eschews zerocopy. > > With 100 flows, even without this patch, 90+% of packets are copied. > Some zerocopy packets from vhost_net fail this test in tun.c > > if (iov_iter_npages(&i, INT_MAX) <= MAX_SKB_FRAGS) > > Generating packets with up to 21 frags. I'm not sure yet why or > what the fraction of these packets is. This seems to be a mix of page alignment and compound pages. The iov_len is always well below the maximum, but frags exceed page size and can start high in the initial page. tun_get_user: num_pages=21 max=17 iov_len=6 len=65226 0: p_off=3264 len=6888 1: p_off=1960 len=16384 2: p_off=1960 len=6232 3: p_off=0 len=10152 4: p_off=1960 len=16384 5: p_off=1960 len=9120 > But this in turn can > disable zcopy_used in vhost_net_tx_select_zcopy for a > larger share of packets: > > return !net->tx_flush && > net->tx_packets / 64 >= net->tx_zcopy_err; > Testing iov_iter_npages() in handle_tx to inform zcopy_used allows skipping these without turning off zerocopy for all other packets. After implementing that, tx_zcopy_err drops to zero, but only around 40% of packets use zerocopy. _______________________________________________ Virtualization mailing list Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/virtualization
