In preparation for adding docs about virtlockd, split out the sanlock setup docs into a separate page. Signed-off-by: Daniel P. Berrange <berrange@xxxxxxxxxx> --- docs/locking-sanlock.html.in | 247 ++++++++++++++++++++++++++++++++++++++ docs/locking.html.in | 277 ++++++------------------------------------- docs/sitemap.html.in | 6 + 3 files changed, 286 insertions(+), 244 deletions(-) create mode 100644 docs/locking-sanlock.html.in diff --git a/docs/locking-sanlock.html.in b/docs/locking-sanlock.html.in new file mode 100644 index 0000000..84b1fb3 --- /dev/null +++ b/docs/locking-sanlock.html.in @@ -0,0 +1,247 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";> +<html xmlns="http://www.w3.org/1999/xhtml";> + <body> + <h1>Virtual machine lock manager, sanlock plugin</h1> + + <ul id="toc"></ul> + + <p> + This page describes use of the + <a href="https://fedorahosted.org/sanlock/";>sanlock</a> + service as a <a href="locking.html">lock driver</a> + plugin for virtual machine disk mutual exclusion. + </p> + + <h2><a name="sanlock">Sanlock daemon setup</a></h2> + + <p> + On many operating systems, the <strong>sanlock</strong> plugin + is distributed in a sub-package which needs to be installed + separately from the main libvirt RPM. On a Fedora/RHEL host + this can be done with the <code>yum</code> command + </p> + + <pre> + $ su - root + # yum install libvirt-lock-sanlock + </pre> + + <p> + The next step is to start the sanlock daemon. For maximum + safety sanlock prefers to have a connection to a watchdog + daemon. This will cause the entire host to be rebooted in + the event that sanlock crashes / terminates abnormally. + To start the watchdog daemon on a Fedora/RHEL host + the following commands can be run: + </p> + + <pre> + $ su - root + # chkconfig wdmd on + # service wdmd start + </pre> + + <p> + Once the watchdog is running, sanlock can be started + as follows + </p> + + <pre> + # chkconfig sanlock on + # service sanlock start + </pre> + + <p> + <em>Note:</em> if you wish to avoid the use of the + watchdog, add the following line to <code>/etc/sysconfig/sanlock</code> + before starting it + </p> + + <pre> + SANLOCKOPTS="-w 0" + </pre> + + <p> + The sanlock daemon must be started on every single host + that will be running virtual machines. So repeat these + steps as necessary. + </p> + + <h2><a name="sanlockplugin">libvirt sanlock plugin configuration</a></h2> + + <p> + Once the sanlock daemon is running, the next step is to + configure the libvirt sanlock plugin. There is a separate + configuration file for each libvirt driver that is using + sanlock. For QEMU, we will edit <code>/etc/libvirt/qemu-sanlock.conf</code> + There is one mandatory parameter that needs to be set, + the <code>host_id</code>. This is a integer between + 1 and 2000, which must be set to a <strong>unique</strong> + value on each host running virtual machines. + </p> + + <pre> + $ su - root + # augtool -s set /files/etc/libvirt/qemu-sanlock.conf/host_id 1 + </pre> + + <p> + Repeat this on every host, changing <strong>1</strong> to a + unique value for the host. + </p> + + <h2><a name="sanlockstorage">libvirt sanlock storage configuration</a></h2> + + <p> + The sanlock plugin needs to create leases in a directory + that is on a filesystem shared between all hosts running + virtual machines. Obvious choices for this include NFS + or GFS2. The libvirt sanlock plugin expects its lease + directory be at <code>/var/lib/libvirt/sanlock</code> + so update the host's <code>/etc/fstab</code> to mount + a suitable shared/cluster filesystem at that location + </p> + + <pre> + $ su - root + # echo "some.nfs.server:/export/sanlock /var/lib/libvirt/sanlock nfs hard,nointr 0 0" >> /etc/fstab + # mount /var/lib/libvirt/sanlock + </pre> + + <p> + If your sanlock daemon happen to run under non-root + privileges, you need to tell this to libvirt so it + chowns created files correctly. This can be done by + setting <code>user</code> and/or <code>group</code> + variables in the configuration file. Accepted values + range is specified in description to the same + variables in <code>/etc/libvirt/qemu.conf</code>. For + example: + </p> + + <pre> + augtool -s set /files/etc/libvirt/qemu-sanlock.conf/user sanlock + augtool -s set /files/etc/libvirt/qemu-sanlock.conf/group sanlock + </pre> + + <p> + But remember, that if this is NFS share, you need a + no_root_squash-ed one for chown (and chmod possibly) + to succeed. + </p> + + <p> + In terms of storage requirements, if the filesystem + uses 512 byte sectors, you need to allow for <code>1MB</code> + of storage for each guest disk. So if you have a network + with 20 virtualization hosts, each running 50 virtual + machines and an average of 2 disks per guest, you will + need <code>20*50*2 == 2000 MB</code> of storage for + sanlock. + </p> + + + <p> + On one of the hosts on the network is it wise to setup + a cron job which runs the <code>virt-sanlock-cleanup</code> + script periodically. This scripts deletes any lease + files which are not currently in use by running virtual + machines, freeing up disk space on the shared filesystem. + Unless VM disks are very frequently created + deleted + it should be sufficient to run the cleanup once a week. + </p> + + <h2><a name="qemuconfig">QEMU/KVM driver configuration</a></h2> + + <p> + The QEMU/KVM driver is fully integrated with the lock + manager framework as of release <span>0.9.3</span>. + The out of the box configuration, however, currently + uses the <strong>nop</strong> lock manager plugin. + To get protection for disks, it is thus necessary + to reconfigure QEMU to activate the <strong>sanlock</strong> + driver. This is achieved by editing the QEMU driver + configuration file (<code>/etc/libvirt/qemu.conf</code>) + and changing the <code>lock_manager</code> configuration + tunable. + </p> + + <pre> + $ su - root + # augtool -s set /files/etc/libvirt/qemu.conf/lock_manager sanlock + # service libvirtd restart + </pre> + + <p> + If all went well, libvirtd will have talked to sanlock + and created the basic lockspace. This can be checked + by looking for existence of the following file + </p> + + <pre> + # ls /var/lib/libvirt/sanlock/ + __LIBVIRT__DISKS__ + </pre> + + <p> + Every time you start a guest, additional lease files will appear + in this directory, one for each virtual disk. The lease + files are named based on the MD5 checksum of the fully qualified + path of the virtual disk backing file. So if the guest is given + a disk backed by <code>/var/lib/libvirt/images/demo.img</code> + expect to see a lease <code>/var/lib/libvirt/sanlock/bfa0240911bc17753e0b473688822159</code> + </p> + + <p> + It should be obvious that for locking to work correctly, every + host running virtual machines should have storage configured + in the same way. The easiest way to do this is to use the libvirt + storage pool capability to configure any NFS volumes, iSCSI targets, + or SCSI HBAs used for guest storage. Simply replicate the same + storage pool XML across every host. It is important that any + storage pools exposing block devices are configured to create + volume paths under <code>/dev/disks/by-path</code> to ensure + stable paths across hosts. An example iSCSI configuration + which ensures this is: + </p> + + <pre> +<pool type='iscsi'> + <name>myiscsipool</name> + <source> + <host name='192.168.254.8'/> + <device path='your-iscsi-target-iqn'/> + </source> + <target> + <path>/dev/disk/by-path</path> + </target> +</pool> + </pre> + + <h2><a name="domainconfig">Domain configuration</a></h2> + + <p> + In case sanlock loses access to disk locks for some reason, it will + kill all domains that lost their locks. This default behavior may + be changed using + <a href="formatdomain.html#elementsEvents">on_lockfailure + element</a> in domain XML. When this element is present, sanlock + will call <code>sanlock_helper</code> (provided by libvirt) with + the specified action. This helper binary will connect to libvirtd + and thus it may need to authenticate if libvirtd was configured to + require that on the read-write UNIX socket. To provide the + appropriate credentials to sanlock_helper, a + <a href="auth.html#Auth_client_config">client authentication + file</a> needs to contain something like the following: + </p> + <pre> +[auth-libvirt-localhost] +credentials=sanlock + +[credentials-sanlock] +authname=login +password=password + </pre> + </body> +</html> diff --git a/docs/locking.html.in b/docs/locking.html.in index 8e1e72c..cf72eb0 100644 --- a/docs/locking.html.in +++ b/docs/locking.html.in @@ -2,258 +2,47 @@ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";> <html xmlns="http://www.w3.org/1999/xhtml";> <body> - <h1>Virtual machine disk locking</h1> + <h1>Virtual machine lock manager</h1> <ul id="toc"></ul> <p> - This page describes how to ensure a single disk cannot be - used by more than one running VM at a time, across any - host in a network. This is critical to avoid data corruption - of guest files systems that are not cluster aware. + Libvirt includes a framework for ensuring mutual exclusion + between virtual machines using host resources. Typically + this is used to prevent two VM processes from having concurrent + write access to the same disk image, as this would result in + data corruption if the guest was not using a cluster + aware filesystem. </p> <h2><a name="plugins">Lock manager plugins</a></h2> <p> - libvirt includes a pluggable framework for lock managers, - which hypervisor drivers can use to ensure safety for - guest domain disks, and potentially other resources. - At this time there are only two plugin implementations, - a "no op" implementation which does absolutely nothing, - and a <a href="https://fedorahosted.org/sanlock/";>sanlock</a> implementation which uses - the Disk Paxos algorithm to ensure safety. - </p> - - <h2><a name="sanlock">Sanlock daemon setup</a></h2> - - <p> - On many operating systems, the <strong>sanlock</strong> plugin - is distributed in a sub-package which needs to be installed - separately from the main libvirt RPM. On a Fedora/RHEL host - this can be done with the <code>yum</code> command - </p> - - <pre> - $ su - root - # yum install libvirt-lock-sanlock - </pre> - - <p> - The next step is to start the sanlock daemon. For maximum - safety sanlock prefers to have a connection to a watchdog - daemon. This will cause the entire host to be rebooted in - the event that sanlock crashes / terminates abnormally. - To start the watchdog daemon on a Fedora/RHEL host - the following commands can be run: - </p> - - <pre> - $ su - root - # chkconfig wdmd on - # service wdmd start - </pre> - - <p> - Once the watchdog is running, sanlock can be started - as follows - </p> - - <pre> - # chkconfig sanlock on - # service sanlock start - </pre> - - <p> - <em>Note:</em> if you wish to avoid the use of the - watchdog, add the following line to <code>/etc/sysconfig/sanlock</code> - before starting it - </p> - - <pre> - SANLOCKOPTS="-w 0" - </pre> - - <p> - The sanlock daemon must be started on every single host - that will be running virtual machines. So repeat these - steps as necessary. - </p> - - <h2><a name="sanlockplugin">libvirt sanlock plugin configuration</a></h2> - - <p> - Once the sanlock daemon is running, the next step is to - configure the libvirt sanlock plugin. There is a separate - configuration file for each libvirt driver that is using - sanlock. For QEMU, we will edit <code>/etc/libvirt/qemu-sanlock.conf</code> - There is one mandatory parameter that needs to be set, - the <code>host_id</code>. This is a integer between - 1 and 2000, which must be set to a <strong>unique</strong> - value on each host running virtual machines. - </p> - - <pre> - $ su - root - # augtool -s set /files/etc/libvirt/qemu-sanlock.conf/host_id 1 - </pre> - - <p> - Repeat this on every host, changing <strong>1</strong> to a - unique value for the host. - </p> - - <h2><a name="sanlockstorage">libvirt sanlock storage configuration</a></h2> - - <p> - The sanlock plugin needs to create leases in a directory - that is on a filesystem shared between all hosts running - virtual machines. Obvious choices for this include NFS - or GFS2. The libvirt sanlock plugin expects its lease - directory be at <code>/var/lib/libvirt/sanlock</code> - so update the host's <code>/etc/fstab</code> to mount - a suitable shared/cluster filesystem at that location - </p> - - <pre> - $ su - root - # echo "some.nfs.server:/export/sanlock /var/lib/libvirt/sanlock nfs hard,nointr 0 0" >> /etc/fstab - # mount /var/lib/libvirt/sanlock - </pre> - - <p> - If your sanlock daemon happen to run under non-root - privileges, you need to tell this to libvirt so it - chowns created files correctly. This can be done by - setting <code>user</code> and/or <code>group</code> - variables in the configuration file. Accepted values - range is specified in description to the same - variables in <code>/etc/libvirt/qemu.conf</code>. For - example: - </p> - - <pre> - augtool -s set /files/etc/libvirt/qemu-sanlock.conf/user sanlock - augtool -s set /files/etc/libvirt/qemu-sanlock.conf/group sanlock - </pre> - - <p> - But remember, that if this is NFS share, you need a - no_root_squash-ed one for chown (and chmod possibly) - to succeed. - </p> - - <p> - In terms of storage requirements, if the filesystem - uses 512 byte sectors, you need to allow for <code>1MB</code> - of storage for each guest disk. So if you have a network - with 20 virtualization hosts, each running 50 virtual - machines and an average of 2 disks per guest, you will - need <code>20*50*2 == 2000 MB</code> of storage for - sanlock. - </p> - - - <p> - On one of the hosts on the network is it wise to setup - a cron job which runs the <code>virt-sanlock-cleanup</code> - script periodically. This scripts deletes any lease - files which are not currently in use by running virtual - machines, freeing up disk space on the shared filesystem. - Unless VM disks are very frequently created + deleted - it should be sufficient to run the cleanup once a week. - </p> - - <h2><a name="qemuconfig">QEMU/KVM driver configuration</a></h2> - - <p> - The QEMU/KVM driver is fully integrated with the lock - manager framework as of release <span>0.9.3</span>. - The out of the box configuration, however, currently - uses the <strong>nop</strong> lock manager plugin. - To get protection for disks, it is thus necessary - to reconfigure QEMU to activate the <strong>sanlock</strong> - driver. This is achieved by editing the QEMU driver - configuration file (<code>/etc/libvirt/qemu.conf</code>) - and changing the <code>lock_manager</code> configuration - tunable. - </p> - - <pre> - $ su - root - # augtool -s set /files/etc/libvirt/qemu.conf/lock_manager sanlock - # service libvirtd restart - </pre> - - <p> - If all went well, libvirtd will have talked to sanlock - and created the basic lockspace. This can be checked - by looking for existence of the following file - </p> - - <pre> - # ls /var/lib/libvirt/sanlock/ - __LIBVIRT__DISKS__ - </pre> - - <p> - Every time you start a guest, additional lease files will appear - in this directory, one for each virtual disk. The lease - files are named based on the MD5 checksum of the fully qualified - path of the virtual disk backing file. So if the guest is given - a disk backed by <code>/var/lib/libvirt/images/demo.img</code> - expect to see a lease <code>/var/lib/libvirt/sanlock/bfa0240911bc17753e0b473688822159</code> - </p> - - <p> - It should be obvious that for locking to work correctly, every - host running virtual machines should have storage configured - in the same way. The easiest way to do this is to use the libvirt - storage pool capability to configure any NFS volumes, iSCSI targets, - or SCSI HBAs used for guest storage. Simply replicate the same - storage pool XML across every host. It is important that any - storage pools exposing block devices are configured to create - volume paths under <code>/dev/disks/by-path</code> to ensure - stable paths across hosts. An example iSCSI configuration - which ensures this is: - </p> - - <pre> -<pool type='iscsi'> - <name>myiscsipool</name> - <source> - <host name='192.168.254.8'/> - <device path='your-iscsi-target-iqn'/> - </source> - <target> - <path>/dev/disk/by-path</path> - </target> -</pool> - </pre> - - <h2><a name="domainconfig">Domain configuration</a></h2> - - <p> - In case sanlock loses access to disk locks for some reason, it will - kill all domains that lost their locks. This default behavior may - be changed using - <a href="formatdomain.html#elementsEvents">on_lockfailure - element</a> in domain XML. When this element is present, sanlock - will call <code>sanlock_helper</code> (provided by libvirt) with - the specified action. This helper binary will connect to libvirtd - and thus it may need to authenticate if libvirtd was configured to - require that on the read-write UNIX socket. To provide the - appropriate credentials to sanlock_helper, a - <a href="auth.html#Auth_client_config">client authentication - file</a> needs to contain something like the following: - </p> - <pre> -[auth-libvirt-localhost] -credentials=sanlock - -[credentials-sanlock] -authname=login -password=password - </pre> + The lock manager framework has a pluggable architecture, + to allow different locking technologies to be used. + </p> + + <dl> + <dt>nop</dt> + <dd>This is a "no op" implementation which does absolutely + nothing. This can be used if mutual exclusion between + virtual machines is not required, or if it is being + solved at another level in the management stack.</dd> + <dt>lockd</dt> + <dd>This is the current preferred implementation shipped + with libvirt. It uses the <code>virtlockd</code> daemon + to manage locks using the POSIX fcntl() advisory locking + capability. As such it requires a shared filesystem of + some kind be accessible to all hosts which share the + same image storage.</dd> + <dt><a href="locking-sanlock.html">sanlock</a></dt> + <dd>This is an alternative implementation preferred by + the oVirt project. It uses a disk paxos algorithm for + maintaining continuously renewed leases. In the default + setup it requires some shared filesystem, but it is + possible to use it in a manual mode where the management + application creates leases in SAN storage volumes. + </dd> + </dl> </body> </html> diff --git a/docs/sitemap.html.in b/docs/sitemap.html.in index fad5358..5b911ce 100644 --- a/docs/sitemap.html.in +++ b/docs/sitemap.html.in @@ -101,6 +101,12 @@ <li> <a href="locking.html">Disk locking</a> <span>Ensuring exclusive guest access to disks</span> + <ul> + <li> + <a href="locking-sanlock.html">Sanlock</a> + <span>Sanlock lock manager plugin</span> + </li> + </ul> </li> <li> <a href="cgroups.html">CGroups</a> -- 2.1.0 -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list