Hi again,
First the good news: I found a way to make live migrations work again.
As quoted below, libvirt changes the ownership of the guest image,
unless it detects that the image is on a shared filesystem. After
looking at the code for libvirt, they have code to detect NFS, GFS2 and
SMB/CIFS, but not Gluster. As libvirt does not detect that the storage
is on a shared file system, the originating host will perform a chown
back to root:root at the end of a successful migration, whereas the
destination host will do a chown to libvirt-qemu:kvm. This is in fact a
race condition, so the difference in behaviour between 3.4.0 and 3.4.1
could be down to timing differences.
Workaround: stop your guests, then stop libvirt, and edit
/etc/libvirt/qemu.conf - this contains a commented out entry
'dynamic_ownership=1', which is the default. Change this to 0, and
remove the comment. Then do a chown to libvirt-qemu:kvm for all your
stopped images. Then you can start the service libvirt-bin again, and
bring up the guests. Repeat on the other half of your cluster, and test
a live migration. For me, they work again.
The workaround seems fine with me, but now you have to take care of
properly setting the ownership of a guest image yourself (presumably
only once when you create it).
Other possible solutions:
JoeJulian suggested using libgfapi, giving libvirt direct access without
having to go through the filesystem. This is the preferred setup for
libvirt+gluster and should also result in better I/O performance. I
haven't tested this yet, but it's high on my to-do list.
Submit a patch to libvirt so it can detect that the filesystem is
Gluster. statfs() will only show 'FUSE", but we could then use getxattr
to see if there is a gluster-specific attribute set (suggested by
kkeithley). This could be trusted.glusterfs.volume-id, e.g.
Regards, Paul Boven.
On 01/28/2014 01:57 PM, Paul Boven wrote:
Hi everyone,
On the libvirt Wiki, I found the text below which might well apply to
our live-migration issue:
"The directory used for storing disk images has to be mounted from
shared storage on both hosts. Otherwise, the domain may lose access to
its disk images during migration because source libvirtd may change the
owner, permissions, and SELinux labels on the disk images once it
successfully migrates the domain to its destination. Libvirt avoids
doing such things if it detects that the disk images are mounted from a
shared storage. "
So perhaps libvirtd fails to recognize that it is on shared storage, and
it is the originating libvirt that throws a wrench in the wheels by
changing the ownership?
http://wiki.libvirt.org/page/Migration_fails_because_disk_image_cannot_be_found
Regards, Paul Boven.
--
Paul Boven <boven@xxxxxxx> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
