The main issue is indeed across nodes. As you pointed out, fnctl() won't solve the issue for iSCSI. What about the leased lock mechanism, for example, with libvirt playing an external watchdog and terminating the qemu process if it cannot renew the lock, or a central (nfs? Central service) storage to maintain the global locks. (would have been very nice if the storage would have played along and accepted ownership assignment/release, but I guess this is too much to ask for) -----Original Message----- From: libvir-list-bounces@xxxxxxxxxx [mailto:libvir-list-bounces@xxxxxxxxxx] On Behalf Of Daniel P. Berrange Sent: Wednesday, October 15, 2008 21:59 PM To: Itamar Heim Cc: libvir-list@xxxxxxxxxx Subject: Re: [libvirt] image locking? On Wed, Oct 15, 2008 at 10:23:03AM -0700, Itamar Heim wrote: > Hi, > > > > I am interested to find out how libvirt envisions image locking. > > i.e., how do we make sure multiple nodes are not trying to access the > same storage volume, probably causing image corruption. > > I know this can be solved by means of a cluster, but it seems excessive > (and not possible in all scenarios). > > Central administration (ovirt-like) is also problematic, since unless > fencing is assumed, it cannot verify the image is no longer being used. > > If the image/storage volume could be locked (or leased lock), either by > the storage/LVM/NFS preventing from a node access to a specific image, > or by having libvirt/qemu mutually check the lock before accessing it > (very simply, in leased lock we can check every X minutes, and kill the > process to make sure it honors the lock). In the domain XML format, the semantics are that every <disk> section added to a guest config is read-write, with an exclusive lock. To allow multiple guests to use the same disk, is intended that you add either <readonly/> or <sharable/> element within the <disk>. That all said, we only implement this for the Xen driver, handing off the actuall logic to XenD to perform. That we don't implement this in the QEMU driver is a clear shortcoming that needs addressing. If we only care about locking within the scope of a single host, it is trivial - libvirtd knows all VMs and their config, so can trivially ensure the appropriate exlusivity checks are done at time of VM start. As you point out, ideally this locking would be enforced across hosts too, in the case of shared storage. Cluster software can't actually magically solve this for us - it can really only make sure the same VM is not started twice. I'm not sure that libivirt can neccessarily solve it in the general case either, but we can at least make an effort in some cases. If, for instance, we were to take a proper fcntl() lock over the files, this would work for disks backed by a file on shared filesystems like NFS / GFS. fcntl() locks won't work on disks backed by iSCSI/SCSI block devices though - and this will actuall play nicely with clustersoftrware too, since they can be made to forcably release NFS locks when fencing a node. It is possible that SCSI reservations can help in the FibreChannel case. So as an immediate option we should perform the exclusivity checks in libvirt, and also apply fcntl() locks over file backed disks. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list -- Libvir-list mailing list Libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list