On Wed, Apr 17, 2019 at 09:09:05 -0500, Eric Blake wrote: > Now that various new API have been added, it is worth a landing page > that gives an overview of capturing various pieces of guest state, and > which APIs are best suited to which tasks. > > Signed-off-by: Eric Blake <eblake@xxxxxxxxxx> > Reviewed-by: John Ferlan <jferlan@xxxxxxxxxx> > Reviewed-by: Daniel P. Berrangé <berrange@xxxxxxxxxx> > --- > docs/docs.html.in | 5 + > docs/domainstatecapture.html.in | 315 ++++++++++++++++++++++++++++++++ > docs/formatbackup.html.in | 4 +- > docs/formatcheckpoint.html.in | 4 +- > docs/formatsnapshot.html.in | 2 + > 5 files changed, 328 insertions(+), 2 deletions(-) > create mode 100644 docs/domainstatecapture.html.in > > diff --git a/docs/docs.html.in b/docs/docs.html.in > index 400b149791..8c08ace402 100644 > --- a/docs/docs.html.in > +++ b/docs/docs.html.in > @@ -124,6 +124,11 @@ > > <dt><a href="secureusage.html">Secure usage</a></dt> > <dd>Secure usage of the libvirt APIs</dd> > + > + <dt><a href="domainstatecapture.html">Domain state > + capture</a></dt> > + <dd>Comparison between different methods of capturing domain > + state</dd> > </dl> > </div> > > diff --git a/docs/domainstatecapture.html.in b/docs/domainstatecapture.html.in > new file mode 100644 > index 0000000000..1d29d70e76 > --- /dev/null > +++ b/docs/domainstatecapture.html.in > @@ -0,0 +1,315 @@ > +<?xml version="1.0" encoding="UTF-8"?> > +<!DOCTYPE html> > +<html xmlns="http://www.w3.org/1999/xhtml";> > + <body> > + > + <h1>Domain state capture using Libvirt</h1> > + > + <ul id="toc"></ul> > + > + <p> > + In order to aid application developers to choose which > + operations best suit their needs, this page compares the > + different means for capturing state related to a domain managed > + by libvirt. > + </p> > + > + <p> > + The information here is primarily geared towards capturing the > + state of an active domain. Capturing the state of an inactive > + domain essentially amounts to copying the contents of guest > + disks, followed by a fresh boot with disks restored to that > + state. > + </p> > + > + <h2><a id="definitions">State capture trade-offs</a></h2> > + > + <p>One of the features made possible with virtual machines is live > + migration -- transferring all state related to the guest from > + one host to another with minimal interruption to the guest's > + activity. In this case, state includes domain memory (including > + register and device contents), and domain storage (whether the > + guest's view of the disks are backed by local storage on the > + host, or by the hypervisor accessing shared storage over a > + network). A clever observer will then note that if all state is > + available for live migration, then there is nothing stopping a > + user from saving some or all of that state at a given point of > + time in order to be able to later rewind guest execution back to > + the state it previously had. The astute reader will also realize > + that state capture at any level requires that the data must be > + stored and managed by some mechanism. This processing might fit > + in a single file, or more likely require a chain of related > + files, and may require synchronization with third-party tools > + built around managing the amount of data resulting from > + capturing the state of multiple guests that each use multiple > + disks. > + </p> > + > + <p> > + There are several libvirt APIs associated with capturing the > + state of a guest, which can later be used to rewind that guest > + to the conditions it was in earlier. The following is a list of > + trade-offs and differences between the various facets that > + affect capturing domain state for active domains: > + </p> > + > + <dl> > + <dt>Duration</dt> > + <dd>Capturing state can be a lengthy process, so while the > + captured state ideally represents an atomic point in time > + corresponding to something the guest was actually executing, > + capturing state tends to focus on minimizing guest downtime > + while performing the rest of the state capture in parallel > + with guest execution. Some interfaces require up-front > + preparation (the state captured is not complete until the API > + ends, which may be some time after the command was first > + started), while other interfaces track the state when the > + command was first issued, regardless of the time spent in > + capturing the rest of the state. Also, time spent in state > + capture may be longer than the time required for live > + migration, when state must be duplicated rather than shared. > + </dd> > + > + <dt>Amount of state</dt> > + <dd>For an online guest, there is a choice between capturing the > + guest's memory (all that is needed during live migration when > + the storage is already shared between source and destination), > + the guest's disk state (all that is needed if there are no > + pending guest I/O transactions that would be lost without the > + corresponding memory state), or both together. Reverting to > + partial state may still be viable, but typically, booting from > + captured disk state without corresponding memory is comparable > + to rebooting a machine that had power cut before I/O could be > + flushed. Guests may need to use proper journaling methods to > + avoid problems when booting from partial state. > + </dd> > + > + <dt>Quiescing of data</dt> > + <dd>Even if a guest has no pending I/O, capturing disk state may > + catch the guest at a time when the contents of the disk are > + inconsistent. Cooperating with the guest to perform data > + quiescing is an optional step to ensure that captured disk > + state is fully consistent without requiring additional memory > + state, rather than just crash-consistent. But guest > + cooperation may also have time constraints, where the guest > + can rightfully panic if there is too much downtime while I/O > + is frozen. > + </dd> > + > + <dt>Quantity of files</dt> > + <dd>When capturing state, some approaches store all state within > + the same file (internal), while others expand a chain of > + related files that must be used together (external), for more > + files that a management application must track. > + </dd> > + > + <dt>Impact to guest definition</dt> > + <dd>Capturing state may require temporary changes to the guest > + definition, such as associating new files into the domain > + definition. While state capture should never impact the > + running guest, a change to the domain's active XML may have > + impact on other host operations being performed on the domain. > + </dd> > + > + <dt>Third-party integration</dt> > + <dd>When capturing state, there are tradeoffs to how much of the > + process must be done directly by the hypervisor, and how much > + can be off-loaded to third-party software. Since capturing > + state is not instantaneous, it is essential that any > + third-party integration see consistent data even if the > + running guest continues to modify that data after the point in > + time of the capture.</dd> > + > + <dt>Full vs. incremental</dt> > + <dd>When periodically repeating the action of state capture, it > + is useful to minimize the amount of state that must be > + captured by exploiting the relation to a previous capture, > + such as focusing only on the portions of the disk that the > + guest has modified in the meantime. Some approaches are able > + to take advantage of checkpoints to provide an incremental > + backup, while others are only capable of a full backup even if > + that means re-capturing unchanged portions of the disk.</dd> > + > + <dt>Local vs. remote</dt> > + <dd>Domains that completely use remote storage may only need > + some mechanism to keep track of guest memory state while using > + external means to manage storage. Still, hypervisor and guest > + cooperation to ensure points in time when no I/O is in flight > + across the network can be important for properly capturing > + disk state.</dd> > + > + <dt>Network latency</dt> > + <dd>Whether it's domain storage or saving domain state into > + remote storage, network latency has an impact on snapshot > + data. Having dedicated network capacity, bandwidth, or quality > + of service levels may play a role, as well as planning for how > + much of the backup process needs to be local.</dd> > + </dl> > + > + <p> > + An example of the various facets in action is migration of a > + running guest. In order for the guest to be able to resume on > + the destination at the same place it left off at the source, the > + hypervisor has to get to a point where execution on the source > + is stopped, the last remaining changes occurring since the > + migration started are then transferred, and the guest is started > + on the target. The management software thus must keep track of > + the starting point and any changes since the starting > + point. These last changes are often referred to as dirty page > + tracking or dirty disk block bitmaps. At some point in time > + during the migration, the management software must freeze the > + source guest, transfer the dirty data, and then start the guest > + on the target. This period of time must be minimal. To minimize > + overall migration time, one is advised to use a dedicated > + network connection with a high quality of service. Alternatively > + saving the current state of the running guest can just be a > + point in time type operation which doesn't require updating the > + "last vestiges" of state prior to writing out the saved state > + file. The state file is the point in time of whatever is current > + and may contain incomplete data which if used to restart the > + guest could cause confusion or problems because some operation > + wasn't completed depending upon where in time the operation was > + commenced. > + </p> > + > + <h2><a id="apis">State capture APIs</a></h2> > + <p>With those definitions, the following libvirt APIs related to > + state capture have these properties:</p> > + <dl> > + <dt><a href="html/libvirt-libvit-domain.html#virDomainManagedSave"><code>virDomainManagedSave</code></a></dt> > + <dd>This API saves guest memory, with libvirt managing all of > + the saved state, then stops the guest. While stopped, the > + disks can be copied by a third party. However, since any > + subsequent restart of the guest by libvirt API will restore > + the memory state (which typically only works if the disk state > + is unchanged in the meantime), and since it is not possible to > + get at the memory state that libvirt is managing, this is not > + viable as a means for rolling back to earlier saved states, > + but is rather more suited to situations such as suspending a > + guest prior to rebooting the host in order to resume the guest > + when the host is back up. This API also has a drawback of > + potentially long guest downtime, and therefore does not lend > + itself well to live backups.</dd> > + > + <dt><a href="html/libvirt-libvit-domain.html#virDomainSave"><code>virDomainSave</code></a></dt> > + <dd>This API is similar to virDomainManagedSave(), but moves the > + burden on managing the stored memory state to the user. As > + such, the user can now couple saved state with copies of the > + disks to perform a revert to an arbitrary earlier saved state. > + However, changing who manages the memory state does not change > + the drawback of potentially long guest downtime when capturing > + state.</dd> > + > + <dt><a href="html/libvirt-libvit-domain-snapshot.html#virDomainSnapshotCreateXML"><code>virDomainSnapshotCreateXML</code></a></dt> > + <dd>This API wraps several approaches for capturing guest state, > + with a general premise of creating a snapshot (where the > + current guest resources are frozen in time and a new wrapper > + layer is opened for tracking subsequent guest changes). It > + can operate on both offline and running guests, can choose > + whether to capture the state of memory, disk, or both when > + used on a running guest, and can choose between internal and > + external storage for captured state. However, it is geared > + towards post-event captures (when capturing both memory and > + disk state, the disk state is not captured until all memory > + state has been collected first). Using QEMU as the > + hypervisor, internal snapshots currently have lengthy downtime > + that is incompatible with freezing guest I/O, but external > + snapshots are quick. Since creating an external snapshot > + changes which disk image resource is in use by the guest, this > + API can be coupled with <a href="html/libvirt-libvirt-domain.html#virDomainBlockCommit"><code>virDomainBlockCommit()</code></a> to Please note here that this is a hack meant to worakround the absence of snapshot reversion/deletion for external snapshots. > + restore things back to the guest using its original disk > + image, where a third-party tool can read the backing file > + prior to the live commit. See also > + the <a href="formatsnapshot.html">XML details</a> used with > + this command.</dd> > + > + <dt><a href="html/libvirt-libvit-domain.html#virDomainFSFreeze"><code>virDomainFSFreeze</code></a>, <a href="html/libvirt-libvit-domain.html#virDomainFSThaw"><code>virDomainFSThaw</code></a></dt> > + <dd>This pair of APIs does not directly capture guest state, but > + can be used to coordinate with a trusted live guest that state > + capture is about to happen, and therefore guest I/O should be > + quiesced so that the state capture is fully consistent, rather > + than merely crash consistent. Some APIs are able to > + automatically perform a freeze and thaw via a flags parameter, > + rather than having to make separate calls to these > + functions. Also, note that freezing guest I/O is only possible > + with trusted guests running a guest agent, and that some > + guests place maximum time limits on how long I/O can be > + frozen.</dd> They technically don't need to be trusted but the guest can then arbitrarily ignore the request. > + > + <dt><a href="html/libvirt-libvit-domain.html#virDomainBlockCopy"><code>virDomainBlockCopy</code></a></dt> > + <dd>This API wraps approaches for capturing the disk state (but > + not memory) of a running guest, but does not track > + accompanying guest memory state, and can only operate on one > + block device per job. To get a consistent copy of multiple > + disks, multiple jobs must be run in parallel, then the domain > + must be paused before ending all of the jobs. The capture is > + consistent only at the end of the operation with a choice for > + future guest changes to either pivot to the new file or to > + resume to just using the original file. The resulting backup > + file is thus the other file no longer in use by the > + guest.</dd> This is somewhat hackish use. The primary use is to change the storage hosting that disk image. > +
Attachment:
signature.asc
Description: PGP signature
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
