On Thu, Aug 08, 2024 at 05:38:03PM -0600, Jim Fehlig via Devel wrote:
> Introduce support for QEMU's new mapped-ram stream format [1].
> mapped-ram is enabled by default if the underlying QEMU advertises
> the mapped-ram migration capability. It can be disabled by changing
> the 'save_image_version' setting in qemu.conf to version '2'.
>
> To use mapped-ram with QEMU:
> - The 'mapped-ram' migration capability must be set to true
> - The 'multifd' migration capability must be set to true and
> the 'multifd-channels' migration parameter must set to 1
> - QEMU must be provided an fdset containing the migration fd
> - The 'migrate' qmp command is invoked with a URI referencing the
> fdset and an offset where to start writing the data stream, e.g.
>
> {"execute":"migrate",
> "arguments":{"detach":true,"resume":false,
> "uri":"file:/dev/fdset/0,offset=0x11921"}}
>
> The mapped-ram stream, in conjunction with direct IO and multifd
> support provided by subsequent patches, can significantly improve
> the time required to save VM memory state. The following tables
> compare mapped-ram with the existing, sequential save stream. In
> all cases, the save and restore operations are to/from a block
> device comprised of two NVMe disks in RAID0 configuration with
> xfs (~8600MiB/s). The values in the 'save time' and 'restore time'
> columns were scraped from the 'real' time reported by time(1). The
> 'Size' and 'Blocks' columns were provided by the corresponding
> outputs of stat(1).
>
> VM: 32G RAM, 1 vcpu, idle (shortly after boot)
>
> | save | restore |
> | time | time | Size | Blocks
> -----------------------+---------+---------+--------------+--------
> legacy | 6.193s | 4.399s | 985744812 | 1925288
> -----------------------+---------+---------+--------------+--------
> mapped-ram | 5.109s | 1.176s | 34368554354 | 1774472
I'm surprised by the restore time speed up, as I didn't think
mapped-ram should make any perf difference without direct IO
and multifd.
> -----------------------+---------+---------+--------------+--------
> legacy + direct IO | 5.725s | 4.512s | 985765251 | 1925328
> -----------------------+---------+---------+--------------+--------
> mapped-ram + direct IO | 4.627s | 1.490s | 34368554354 | 1774304
Still somewhat surprised by the speed up on restore here too
> -----------------------+---------+---------+--------------+--------
> mapped-ram + direct IO | | | |
> + multifd-channels=8 | 4.421s | 0.845s | 34368554318 | 1774312
> -------------------------------------------------------------------
>
> VM: 32G RAM, 30G dirty, 1 vcpu in tight loop dirtying memory
>
> | save | restore |
> | time | time | Size | Blocks
> -----------------------+---------+---------+--------------+---------
> legacy | 25.800s | 14.332s | 33154309983 | 64754512
> -----------------------+---------+---------+--------------+---------
> mapped-ram | 18.742s | 15.027s | 34368559228 | 64617160
> -----------------------+---------+---------+--------------+---------
> legacy + direct IO | 13.115s | 18.050s | 33154310496 | 64754520
> -----------------------+---------+---------+--------------+---------
> mapped-ram + direct IO | 13.623s | 15.959s | 34368557392 | 64662040
These figures make more sense with restore time matching save time
more or less.
> -----------------------+-------- +---------+--------------+---------
> mapped-ram + direct IO | | | |
> + multifd-channels=8 | 6.994s | 6.470s | 34368554980 | 64665776
> --------------------------------------------------------------------
>
> As can be seen from the tables, one caveat of mapped-ram is the logical
> file size of a saved image is basically equivalent to the VM memory size.
> Note however that mapped-ram typically uses fewer blocks on disk.
>
> Another caveat of mapped-ram is the requirement for a seekable file
> descriptor, which currently makes it incompatible with libvirt's
> support for save image compression. Also note the mapped-ram stream
> is incompatible with the existing stream format, hence mapped-ram
> cannot be used to restore an image saved with the existing format
> and vice versa.
>
> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/mapped-ram.rst?ref_type=heads
>
> Signed-off-by: Jim Fehlig <jfehlig@xxxxxxxx>
> ---
> src/qemu/qemu_driver.c | 20 ++++--
> src/qemu/qemu_migration.c | 139 ++++++++++++++++++++++++++------------
> src/qemu/qemu_migration.h | 4 +-
> src/qemu/qemu_monitor.c | 36 ++++++++++
> src/qemu/qemu_monitor.h | 4 ++
> src/qemu/qemu_saveimage.c | 43 +++++++++---
> src/qemu/qemu_saveimage.h | 2 +
> src/qemu/qemu_snapshot.c | 9 ++-
> 8 files changed, 195 insertions(+), 62 deletions(-)
>
> diff --git a/src/qemu/qemu_saveimage.c b/src/qemu/qemu_saveimage.c
> index 6f2ce40124..98a1ad638d 100644
> --- a/src/qemu/qemu_saveimage.c
> +++ b/src/qemu/qemu_saveimage.c
> @@ -96,6 +96,7 @@ G_DEFINE_AUTOPTR_CLEANUP_FUNC(virQEMUSaveData, virQEMUSaveDataFree);
> */
> virQEMUSaveData *
> virQEMUSaveDataNew(virQEMUDriver *driver,
> + virDomainObj *vm,
> char *domXML,
> qemuDomainSaveCookie *cookieObj,
> bool running,
> @@ -115,6 +116,19 @@ virQEMUSaveDataNew(virQEMUDriver *driver,
> header = &data->header;
> memcpy(header->magic, QEMU_SAVE_PARTIAL, sizeof(header->magic));
> header->version = cfg->saveImageVersion;
> +
> + /* Enable mapped-ram feature if available and save version >= 3 */
> + if (header->version >= QEMU_SAVE_VERSION &&
> + qemuMigrationCapsGet(vm, QEMU_MIGRATION_CAP_MAPPED_RAM)) {
> + if (compressed != QEMU_SAVE_FORMAT_RAW) {
> + virReportError(VIR_ERR_OPERATION_FAILED,
> + _("compression is not supported with save image version %1$u"),
> + header->version);
> + goto error;
> + }
> + header->features |= QEMU_SAVE_FEATURE_MAPPED_RAM;
> + }
If the QEMU we're usnig doesnt have CAP_MAPPED_RAM, then I think
we should NOT default to Version 3 save images, as that's creating
a backcompat problem for zero user benefit.
This suggests that in qemu_conf.c, we should initialize the
default value to '0', and then in this code, if we see
version 0 we should pick either 2 or 3 depending on mapped
ram.
> +
> header->was_running = running ? 1 : 0;
> header->compressed = compressed;
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
