On Fri, Sep 29, 2017 at 09:25:27PM -0400, Willem de Bruijn wrote: > On Fri, Sep 29, 2017 at 3:38 PM, Michael S. Tsirkin <mst@xxxxxxxxxx> wrote: > > On Wed, Sep 27, 2017 at 08:25:56PM -0400, Willem de Bruijn wrote: > >> From: Willem de Bruijn <willemb@xxxxxxxxxx> > >> > >> Vhost-net has a hard limit on the number of zerocopy skbs in flight. > >> When reached, transmission stalls. Stalls cause latency, as well as > >> head-of-line blocking of other flows that do not use zerocopy. > >> > >> Instead of stalling, revert to copy-based transmission. > >> > >> Tested by sending two udp flows from guest to host, one with payload > >> of VHOST_GOODCOPY_LEN, the other too small for zerocopy (1B). The > >> large flow is redirected to a netem instance with 1MBps rate limit > >> and deep 1000 entry queue. > >> > >> modprobe ifb > >> ip link set dev ifb0 up > >> tc qdisc add dev ifb0 root netem limit 1000 rate 1MBit > >> > >> tc qdisc add dev tap0 ingress > >> tc filter add dev tap0 parent ffff: protocol ip \ > >> u32 match ip dport 8000 0xffff \ > >> action mirred egress redirect dev ifb0 > >> > >> Before the delay, both flows process around 80K pps. With the delay, > >> before this patch, both process around 400. After this patch, the > >> large flow is still rate limited, while the small reverts to its > >> original rate. See also discussion in the first link, below. > >> > >> The limit in vhost_exceeds_maxpend must be carefully chosen. When > >> vq->num >> 1, the flows remain correlated. This value happens to > >> correspond to VHOST_MAX_PENDING for vq->num == 256. Allow smaller > >> fractions and ensure correctness also for much smaller values of > >> vq->num, by testing the min() of both explicitly. See also the > >> discussion in the second link below. > >> > >> Link:http://lkml.kernel.org/r/CAF=yD-+Wk9sc9dXMUq1+x_hh=3ThTXa6BnZkygP3tgVpjbp93g@xxxxxxxxxxxxxx > >> Link:http://lkml.kernel.org/r/20170819064129.27272-1-den@xxxxxxxxxxxxx > >> Signed-off-by: Willem de Bruijn <willemb@xxxxxxxxxx> > > > > I'd like to see the effect on the non rate limited case though. > > If guest is quick won't we have lots of copies then? > > Yes, but not significantly more than without this patch. > > I ran 1, 10 and 100 flow tcp_stream throughput tests from a sender > in the guest to a receiver in the host. > > To answer the other benchmark question first, I did not see anything > noteworthy when increasing vq->num from 256 to 1024. > > With 1 and 10 flows without this patch all packets use zerocopy. > With the patch, less than 1% eschews zerocopy. > > With 100 flows, even without this patch, 90+% of packets are copied. > Some zerocopy packets from vhost_net fail this test in tun.c > > if (iov_iter_npages(&i, INT_MAX) <= MAX_SKB_FRAGS) > > Generating packets with up to 21 frags. I'm not sure yet why or > what the fraction of these packets is. But this in turn can > disable zcopy_used in vhost_net_tx_select_zcopy for a > larger share of packets: > > return !net->tx_flush && > net->tx_packets / 64 >= net->tx_zcopy_err; > > Because the number of copied and zerocopy packets are the > same before and after the patch, so are the overall throughput > numbers. OK, thanks! Are you looking into new warnings that kbuild system reported with this patch? Thanks, -- MST _______________________________________________ Virtualization mailing list Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/virtualization
