On Mon, Jul 1, 2019 at 7:56 PM Daniel P. Berrangé <berrange@xxxxxxxxxx> wrote: > > On Mon, Jul 01, 2019 at 05:26:45PM +0200, Christian Ehrhardt wrote: > > Hi, > > today I was debugging an issue that I found with qemu 4.0 and ended up > > puzzled about file ownership. I'm almost EOD now, but wanted to reach > > out here for a sanity check before I debug further. The case > > sumamrized is this: > > > > 1. start guest > > 1.1 image files are changed to libvirt-qemu:kvm (which matches > > Ubuntus user/group config) > > 2. live migrate the guest to a different node > > 2.1 image files go back to root:root which they initially had (ok) > > 3. migrate guest back to the original node > > x. image files stay root:root and are not changed back to libvirt-qemu:kvm > > > > That is odd/unexpected, but it seems the same applies to older > > versions and there it was never a problem so far. > > This sounds odd - I see no reason why the first migration should > behave differently from the second migration, as libvirt makes > no distinction & has no knowledge of previous migrations. I might have been unclear, let me clarify: 1. migration without copy (changes the ownership) lxc exec testkvm-eoan-from -- virsh migrate --live --unsafe kvmguest-eoan-normal qemu+ssh://10.222.144.19/system (and the same type backwards) 2. migration away from here with --copy-storage* fails due to file ownership > Could the two hosts be configured differently in some way. For > example is the disk image on ext4 on one host, but nfs on the > other host ? With shared filesystems, we'd generally expect > the disk to be exposed on a shared filesystem on both hosts. The two hosts are in fact two LXD containers sharing the same Filesystem (for the first migration) And still two containers, but without shared FS for the second copy-storage-* migration that then eventually fails due to the bad file ownership. But since they in fact use shared storage that might still be the reason - maybe the ordering is important here. Since in the setup for migration #1 they really use shared storage it might be that we have something like 1. source: migrates off guest 2. target: receives guest 3. target: guest is complete and changes file ownership to libvirt-qemu:kvm (correct) 4. source: shuts down the stub that is left after migration and changes file ownership to root:root Due to really using the same storage (not like the same FCP or scsi device but the same FS) that could be the reason the ownership after a migration cycle is reset. Thanks for the hint, worth a check with some slightly altered setups ... ... and confirmed - if I use copied images, but not on a shared FS the ownership is handled correctly. So my way of sharing the FS might be odd. And libvirt does not detect it as such, and due to that above ordering triggers the issue or the ownership being set to root:root by the migration source after the migration is complete. Quoting the virsh man page my setup is similar to "disk images are stored on coherent clustered filesystem, such as GFS2 or GPFS" but my libvirt doesn't know that and therefore changes the ownership. Maybe it would have the same bug on GFS2/GPFS? I don't have such a setup at hand, how is the issue avoided there? Is there a way to make libvirt realize that it really is on "the same" FS to avoid these operations? > We do take special action if we see an NFS filesystem, as we > cannot reset file ownership on the migration source, as that > would impact the migration dest too. > > > In my case I stumbled over it because the newer qemu --copy-storage-* > > options now want to re-open the file at some point and that fails > > with: > > error: internal error: unable to execute QEMU command 'drive-mirror': > > Could not reopen file: Permission denied > > > > If I e.g. shut down and start the guest on the source node (which > > fixes up the ownership at runtime) then migration with copy-storage > > works fine again. > > > > I'm mostly looking for hints if this is a known issue or being worked > > on to not waste time tomorrow. > > So hints are welcome. > > > > -- > > Christian Ehrhardt > > Software Engineer, Ubuntu Server > > Canonical Ltd > > > > -- > > libvir-list mailing list > > libvir-list@xxxxxxxxxx > > https://www.redhat.com/mailman/listinfo/libvir-list > > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
