On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote:
> On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
> > I tried another workaround which was to get virt-resize to fsync the
> > output file before closing the libvirt connection, but that doesn't
> > work for reasons I don't understand so far - still studying this.
>
> I worked out what was happening here -- I'd inserted the fsync at the
> wrong place in virt-resize. So I have now successfully worked around
> this for the virt-resize case, however it's still a problem that could
> manifest itself in other uses of libvirt + qemu + slow devices.
We've seen the "Failed to terminate process 1275 with SIGTERM: Device or
resource busy" error occur in context of OpenStack as well[1][2].
The behavior is from virDomainDestroy() API (src/libvirt-domain.c):
[...]
* virDomainDestroy first requests that a guest terminate (e.g.
* SIGTERM), then waits for it to comply. After a reasonable timeout,
* if the guest still exists, virDomainDestroy will forcefully
* terminate the guest (e.g. SIGKILL) if necessary (which may produce
* undesirable results, for example unflushed disk cache in the
* guest). To avoid this possibility, it's recommended to instead
* call virDomainDestroyFlags, sending the
* VIR_DOMAIN_DESTROY_GRACEFUL flag.
[...]
Dan Berrange explains[1]:
There are two reasons why you'd get this failure ("Failed to terminate
process: Device or resource busy") from libvirt.
- The host is so overloaded that the kernel was not able to clean up
the process in the time that libvirt was prepared to wait. If this
is the case, the process should eventually go away on its own
after a short while longer and everything should return to normal
- There is some problem, causing the process to get stuck in an
uninterruptable wait state. This is usually due to something going
wrong in the storage stack, causing some I/O read/write operation
to hang in kernel space. In this case the process will stay around
in the zombie state forever, or until the storage problem is
resolved.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1205647 --
nova.virt.libvirt.driver fails to shutdown reboot instance with
error 'Code=38 Error=Failed to terminate process 4260 with SIGKILL:
Device or resource busy'
[2] https://bugs.launchpad.net/nova/+bug/1353939 -- Rescue fails with
'Failed to terminate process: Device or resource busy' in the n-cpu
log
--
/kashyap
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list