kvm-owner@xxxxxxxxxxxxxxx wrote on 29/07/2014 04:30:34 AM: > From: "Zhang Haoyu" <zhanghy@xxxxxxxxxxx> > To: "Jason Wang" <jasowang@xxxxxxxxxx>, "Abel Gordon" > <abel.gordon@xxxxxxxxx>, > Cc: Razya Ladelsky/Haifa/IBM@IBMIL, Alex Glikson/Haifa/IBM@IBMIL, > Eran Raichstein/Haifa/IBM@IBMIL, Joel Nider/Haifa/IBM@IBMIL, "kvm" > <kvm@xxxxxxxxxxxxxxx>, "Michael S. Tsirkin" <mst@xxxxxxxxxx>, Yossi > Kuperman1/Haifa/IBM@IBMIL > Date: 29/07/2014 04:35 AM > Subject: Re: [PATCH] vhost: Add polling mode > Sent by: kvm-owner@xxxxxxxxxxxxxxx > > Maybe tie a knot between "vhost-net scalability tuning: threading > for many VMs" and "vhost: Add polling mode" is a good marriage, > because it's more possibility to get work to do with less polling > time, so less cpu cycles waste. > Hi Zhang, Indeed have one vhost thread shared by multiple vms, polling for their requests is the ultimate goal of this plan. The current challenge with it is that the cgroup mechanism needs to be supported/incorporated somehow by this shared vhost thread, as it now serves multiple vms(processes). B.T.W. - if someone wants to help with this effort (mainly the cgroup issue), it would be greatly appreciated...! Thank you, Razya > Thanks, > Zhang Haoyu > > >>>>>> > >>> Hello All, > >>>>>> > >>> > >>>>>> > >>> When vhost is waiting for buffers from the guest driver > (e.g., more > >>>>>> > >>> packets > >>>>>> > >>> to send in vhost-net's transmit queue), it normally > goes to sleep and > >>>>>> > >>> waits > >>>>>> > >>> for the guest to "kick" it. This kick involves a PIO in > the guest, and > >>>>>> > >>> therefore an exit (and possibly userspace involvement > in translating > >>>> > > this > >>>>>> > >>> PIO > >>>>>> > >>> exit into a file descriptor event), all of which hurts > performance. > >>>>>> > >>> > >>>>>> > >>> If the system is under-utilized (has cpu time to > spare), vhost can > >>>>>> > >>> continuously poll the virtqueues for new buffers, and > avoid asking > >>>>>> > >>> the guest to kick us. > >>>>>> > >>> This patch adds an optional polling mode to vhost, that > can be enabled > >>>>>> > >>> via a kernel module parameter, "poll_start_rate". > >>>>>> > >>> > >>>>>> > >>> When polling is active for a virtqueue, the guest is asked to > >>>>>> > >>> disable notification (kicks), and the worker thread continuously > >>>> > > checks > >>>>>> > >>> for > >>>>>> > >>> new buffers. When it does discover new buffers, it > simulates a "kick" > >>>> > > by > >>>>>> > >>> invoking the underlying backend driver (such as vhost-net), which > >>>> > > thinks > >>>>>> > >>> it > >>>>>> > >>> got a real kick from the guest, and acts accordingly. If the > >>>> > > underlying > >>>>>> > >>> driver asks not to be kicked, we disable polling on > this virtqueue. > >>>>>> > >>> > >>>>>> > >>> We start polling on a virtqueue when we notice it has > >>>>>> > >>> work to do. Polling on this virtqueue is later disabled after 3 > >>>> > > seconds of > >>>>>> > >>> polling turning up no new work, as in this case we are better off > >>>>>> > >>> returning > >>>>>> > >>> to the exit-based notification mechanism. The default > timeout of 3 > >>>> > > seconds > >>>>>> > >>> can be changed with the "poll_stop_idle" kernel module parameter. > >>>>>> > >>> > >>>>>> > >>> This polling approach makes lot of sense for new HW with > >>>> > > posted-interrupts > >>>>>> > >>> for which we have exitless host-to-guest notifications. > But even with > >>>>>> > >>> support > >>>>>> > >>> for posted interrupts, guest-to-host communication > still causes exits. > >>>>>> > >>> Polling adds the missing part. > >>>>>> > >>> > >>>>>> > >>> When systems are overloaded, there won?t be enough cpu > time for the > >>>>>> > >>> various > >>>>>> > >>> vhost threads to poll their guests' devices. For these > scenarios, we > >>>> > > plan > >>>>>> > >>> to add support for vhost threads that can be shared by multiple > >>>> > > devices, > >>>>>> > >>> even of multiple vms. > >>>>>> > >>> Our ultimate goal is to implement the I/O acceleration features > >>>> > > described > >>>>>> > >>> in: > >>>>>> > >>> KVM Forum 2013: Efficient and Scalable Virtio (by Abel Gordon) > >>>>>> > >>> https://www.youtube.com/watch?v=9EyweibHfEs > >>>>>> > >>> and > >>>>>> > >>> https://www.mail-archive.com/kvm@xxxxxxxxxxxxxxx/msg98179.html > >>>>>> > >>> > >>>>>> > >>> > >>>>>> > >>> Comments are welcome, > >>>>>> > >>> Thank you, > >>>>>> > >>> Razya > >>>>> > >> Thanks for the work. Do you have perf numbers for this? > >>>>> > >> > >>>> > > Hi Jason, > >>>> > > Thanks for reviewing. I ran some experiments with TCP > stream netperf and > >>>> > > filebench (having 2 threads performing random reads) > benchmarks on an IBM > >>>> > > System x3650 M4. > >>>> > > All runs loaded the guests in a way that they were (cpu) saturated. > >>>> > > The system had two cores per guest, as to allow for both > the vcpu and the > >>>> > > vhost thread to > >>>> > > run concurrently for maximum throughput (but I didn't pin > the threads to > >>>> > > specific cores) > >>>> > > I get: > >>>> > > > >>>> > > Netperf, 1 vm: > >>>> > > The polling patch improved throughput by ~33%. Number of exits/sec > >>>> > > decreased 6x. > >>>> > > The same improvement was shown when I tested with 3 vms > running netperf. > >>>> > > > >>>> > > filebench, 1 vm: > >>>> > > ops/sec improved by 13% with the polling patch. Number of exits was > >>>> > > reduced by 31%. > >>>> > > The same experiment with 3 vms running filebench showed > similar numbers. > >>> > > >>> > Looks good, may worth to add the result in the commit log. > >>>> > > > >>>>> > >> And looks like the patch only poll for virtqueue. In the > future, may > >>>>> > >> worth to add callbacks for vhost_net to poll socket. Then > it could be > >>>>> > >> used with rx busy polling in host which may speedup the rx also. > >>>> > > Did you mean polling the network device to avoid interrupts? > >>> > > >>> > Yes, recent linux host support rx busy polling which can reduce the > >>> > interrupts. If vhost can utilize this, it can also reduce the latency > >>> > caused by vhost thread wakeups. > >>> > > >>> > And I'm also working on virtio-net busy polling in guest, if vhost can > >>> > poll socket, it can also help in guest rx polling. > >> Nice :) Note that you may want to check if if the processor support > >> posted interrupts. I guess that if CPU supports posted interrupts then > >> benefits of polling in the front-end (from performance perspective) > >> may not worth the cpu cycles wasted in the guest. > >> > > > >Yes it's worth to check. But I think busy polling in guest may still > >help since it may reduce the overhead of irq and NAPI in guest, also can > >reduce the latency by eliminating wakeups of both vcpu thread in host > >and userspace process in guest. > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
