On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
As people may know, we frequently encounter errors caused by libvirt when running the libguestfs appliance. I wanted to find out exactly how frequently these happen and classify the errors, so I ran the 'virt-df' tool overnight 1700 times. This tool runs several parallel qemu:///session libvirt connections both creating a short-lived appliance guest. Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183 "XML-RPC error : Cannot write data: Transport endpoint is not connected" Results: The test failed 538 times (32% of the time), which is pretty dismal. To be fair, virt-df is aggressive about how it launches parallel libvirt connections. Most other virt-* tools use only a single libvirt connection and are consequently more reliable. Of the failures, 518 (96%) were of the form: process exited while connecting to monitor: qemu: could not load kernel '/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission denied which is https://bugzilla.redhat.com/921135 or maybe https://bugzilla.redhat.com/1269975. It's not clear to me if these bugs have different causes, but if they do then potentially we're seeing a mix of both since my test has no way to distinguish them.
It looks to me as the same problem. And as the same problem we were talking about bunch of time and, apparently, didn't get to a conclusion. For each of the kernels, libvirt labels them (with both DAC and selinux labels), then proceeds to launching qemu. If this is done parallel, the race is pretty obvious. Could you remind me why you couldn't use <seclabel model='none'/> or <seclabel relabel='no'/> or something that would mitigate this? If we cannot use this, then we need to implement the <seclabel/> element for kernel and initrd.
19 of the failures (4%) were of the form: process exited while connecting to monitor: fread() failed which I believe is a previously unknown bug. I have filed it as https://bugzilla.redhat.com/1298122
I think even this one might be the case, maybe selinux stops qemu from reading the kernel/initrd.
Finally there was 1 failure: Unable to read from monitor: Connection reset by peer which I believe is also a new bug. I have filed it as https://bugzilla.redhat.com/1298124
This, I believe, means QEMU exited (as in the previous one), just at different point in time.
I would be good if libvirt could routinely test the case of multiple parallel launches of qemu:///session, since it still contains bugs even after Cole's fixes. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
Attachment:
signature.asc
Description: PGP signature
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list