On Mon, Jan 16, 2017 at 10:00 AM, Karl Rister <krister@xxxxxxxxxx> wrote: > On 01/16/2017 07:15 AM, Stefan Hajnoczi wrote: >> On Tue, Jan 03, 2017 at 12:02:14PM -0500, Weiwei Jia wrote: >>>> The expensive part is the virtqueue kick. Recently we tried polling the >>>> virtqueue instead of waiting for the ioeventfd file descriptor and got >>>> double-digit performance improvements: >>>> https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg00148.html >>>> >>>> If you want to understand the performance of your benchmark you'll have >>>> compare host/guest disk stats (e.g. request lifetime, disk utilization, >>>> queue depth, average request size) to check that the bare metal and >>>> guest workloads are really sending comparable I/O patterns to the >>>> physical disk. >>>> >>>> Then you using Linux and/or QEMU tracing to analyze the request latency >>>> by looking at interesting points in the request lifecycle like virtqueue >>>> kick, host Linux AIO io_submit(2), etc. >>>> >>> >>> Thank you. I will look into "polling the virtqueue" as you said above. >>> Currently, I just use blktrace to see disk stats and add logs in the >>> I/O workload to see the time latency for each request. What kind of >>> tools are you using to analyze request lifecycle like virtqueue kick, >>> host Linux AIO iosubmit, etc. >>> >>> Do you trace the lifecycle like this >>> (http://www.linux-kvm.org/page/Virtio/Block/Latency#Performance_data) >>> but it seems to be out of date. Does it >>> (http://repo.or.cz/qemu-kvm/stefanha.git/shortlog/refs/heads/tracing-dev-0.12.4) >>> still work on QEMU 2.4.1? >> >> The details are out of date but the general approach to tracing the I/O >> request lifecycle still apply. >> >> There are multiple tracing tools that can do what you need. I've CCed >> Karl Rister who did the latest virtio-blk dataplane tracing. > > I roughly followed this guide by Luiz Capitulino: > > https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html > > I tweaked his trace-host-and-guest script to avoid doing IO while > tracing is enabled, my version is available here: > > http://people.redhat.com/~krister/tracing/trace-host-and-guest > > I built QEMU with --enable-trace-backends=ftrace and then turned on the > QEMU trace events I was interested in with this bit of bash: > > for event in $(/usr/libexec/qemu-kvm -trace help 2>&1|grep virtio|grep > -v "gpu\|console\|serial\|rng\|balloon\|ccw"); do virsh > qemu-monitor-command master --hmp trace-event ${event} on; done > > At this point, the QEMU trace events are automatically inserted into the > ftrace buffers and the methodology outlined by Luiz gets the guest > kernel, host kernel, and QEMU events properly interleaved. Thank you. I will try it. Best, Weiwei Jia -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
