On 01/16/2017 07:15 AM, Stefan Hajnoczi wrote: > On Tue, Jan 03, 2017 at 12:02:14PM -0500, Weiwei Jia wrote: >>> The expensive part is the virtqueue kick. Recently we tried polling the >>> virtqueue instead of waiting for the ioeventfd file descriptor and got >>> double-digit performance improvements: >>> https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg00148.html >>> >>> If you want to understand the performance of your benchmark you'll have >>> compare host/guest disk stats (e.g. request lifetime, disk utilization, >>> queue depth, average request size) to check that the bare metal and >>> guest workloads are really sending comparable I/O patterns to the >>> physical disk. >>> >>> Then you using Linux and/or QEMU tracing to analyze the request latency >>> by looking at interesting points in the request lifecycle like virtqueue >>> kick, host Linux AIO io_submit(2), etc. >>> >> >> Thank you. I will look into "polling the virtqueue" as you said above. >> Currently, I just use blktrace to see disk stats and add logs in the >> I/O workload to see the time latency for each request. What kind of >> tools are you using to analyze request lifecycle like virtqueue kick, >> host Linux AIO iosubmit, etc. >> >> Do you trace the lifecycle like this >> (http://www.linux-kvm.org/page/Virtio/Block/Latency#Performance_data) >> but it seems to be out of date. Does it >> (http://repo.or.cz/qemu-kvm/stefanha.git/shortlog/refs/heads/tracing-dev-0.12.4) >> still work on QEMU 2.4.1? > > The details are out of date but the general approach to tracing the I/O > request lifecycle still apply. > > There are multiple tracing tools that can do what you need. I've CCed > Karl Rister who did the latest virtio-blk dataplane tracing. I roughly followed this guide by Luiz Capitulino: https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html I tweaked his trace-host-and-guest script to avoid doing IO while tracing is enabled, my version is available here: http://people.redhat.com/~krister/tracing/trace-host-and-guest I built QEMU with --enable-trace-backends=ftrace and then turned on the QEMU trace events I was interested in with this bit of bash: for event in $(/usr/libexec/qemu-kvm -trace help 2>&1|grep virtio|grep -v "gpu\|console\|serial\|rng\|balloon\|ccw"); do virsh qemu-monitor-command master --hmp trace-event ${event} on; done At this point, the QEMU trace events are automatically inserted into the ftrace buffers and the methodology outlined by Luiz gets the guest kernel, host kernel, and QEMU events properly interleaved. > > "perf record -a -e kvm:\*" is a good start. You can use "perf probe" to > trace QEMU's trace events (recent versions have sdt support, which means > SystemTap tracepoints work) and also trace any function in QEMU: > http://blog.vmsplice.net/2011/03/how-to-use-perf-probe.html > > Stefan > -- Karl Rister <krister@xxxxxxxxxx> -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
