Re: [libvirt RFC] add API for parallel Saves (not for committing)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Apr 25, 2022 at 03:25:43PM +0100, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrangé (berrange@xxxxxxxxxx) wrote:
> > On Mon, Apr 25, 2022 at 01:33:41PM +0100, Dr. David Alan Gilbert wrote:
> > > * Daniel P. Berrangé (berrange@xxxxxxxxxx) wrote:
> > > > On Mon, Apr 25, 2022 at 12:04:37PM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Daniel P. Berrangé (berrange@xxxxxxxxxx) wrote:
> > > > > > I think we need to introduce an explicit 'file:' protocol
> > > > > > for the migrate command, that is backed by the blockdev APIs
> > > > > > so it can do O_DIRECT and non-blocking AIO.  For the 'fd:'
> > > > > > protocol, we need to be able to tell QEMU whether the 'fd'
> > > > > > is a stream or a regular file, so it can choose between the
> > > > > > regular send/recv APIs, vs the Blockdev APIs (maybe we can
> > > > > > auto-detect with fstat()).  If we do this, then multifd
> > > > > > doesn't end up needing multiple save files on disk, all
> > > > > > the threads can be directly writing to the same file, just
> > > > > > as the relevant offsets on disk to match the RAM page
> > > > > > location.
> > > > > 
> > > > > Hmm so what I'm not sure of is whether it makes sense to use the normal
> > > > > migration flow/code for this or not; and you're suggesting a few
> > > > > possibly contradictory things.
> > > > > 
> > > > > Adding a file: protocol would be pretty easy (whether it went via
> > > > > the blockdev layer or not); getting it to be more efficient is the
> > > > > tricky part, because we've got loads of levels of stream abstraction in
> > > > > the RAM save code:
> > > > >     QEMUFile->channel->OS
> > > > > but then if you want to enforce alignment you somehow have to make that
> > > > > go all the way down.
> > > > 
> > > > The QIOChannel stuff doesn't add buffering, so I wasn't worried
> > > > about alignment there.
> > > > 
> > > > QEMUFile has optional buffering which would mess with alignment,
> > > > but we could turn that off potentially for the RAM transfer, if
> > > > using multifd.
> > > 
> > > The problem isn't whether they add buffering or not; the problem is you
> > > now need a way to add a mechanism to ask for alignment.
> > > 
> > > > I'm confident the performance on the QMEU side though could
> > > > exceed what's viable with libvirt's iohelper  today, as we
> > > > would definitely be eliminating 1 copy and many context switches.
> > > 
> > > Yes but you get that just from adding a simple file: (or fd:) mode
> > > without trying to do anything clever with alignment or rewriting the
> > > same offset.
> > 
> > I don't think so, as libvirt supports O_DIRECT today to avoid
> > trashing the host cache when saving VMs. So to be able to
> > offload libvirt's work to QEMU, O_DIRECT is a pre-requisite.
> 
> I guess you could O_DIRECT it from a buffer in QemuFile or the channel.
> 
> > So we do need the alignment support at the very least. Rewriting
> > at the same offset isnt mandatory, but I think it'd make multifd
> > saner if trying to have all threads work on the same file.
> 
> Thinking on the fly, you'd need some non trivial changes:
> 
>   a) A section entry in the format to say 'align to ... n bytes'
>     (easyish)

Yep

>   b) A way to allocate a location in the file to a RAMBlock
>     [ We already have a bitmap address, so that might do,  but
>     you need to make it interact with the existing file, so it might
>     be easier to do the allocate and record it ]

IIUC, the migration protocol first serializes all RAM, and then serializes
the VMstate for devices.  When libvirt creates a save image for a VM it
has its own header + XML dump and then appends the migrate stream.

So we get a layout of

   +------------------+
   | libvirt header   |
   +------------------+
   | libvirt XML      |
   | ...              |
   +------------------+
   | migration stream |
   | ...              |
   +------------------+

The 'migration stream' is opaque as far as libvirt is concerned,
but we happen to know that from QEMU POV it expands to

   +------------------+
   | RAM stream       |
   | ...              |
   +------------------+
   | vmstate          |
   | ...              |
   +------------------+

Where 'RAM stream' is a stream of the RAM block header and
RAM block contents, for every page.

In the suggestion above, my desire is to achieve this layout


   +------------------+
   | libvirt header   |
   +------------------+
   | libvirt XML      |
   | ...              |
   +------------------+
   | RAM              |
   | ...              |
   +------------------+
   | vmstate          |
   | ...              |
   +------------------+

The key difference being 'RAM' instead of 'RAM stream'. Libvirt
would have to tell QEMU what offset in the file it is permitted
to start at - say 16 MB.  'RAM' would be a 1:1 mapping of the
guest RAM, simply offset by that 16 MB.

IOW, I'm assuming that a 4 GB RAM VM would have its RAM written
starting from offfset  16 MB and ending at 4 GB + 16 MB.

I'm thinking stuff like virtio-mem / RAM hotplug makes life
harder though as there can be many distinct blocks of RAM
contributing to a QEMU map, and we need to be able to declare
an ordering of them, for mapping to the file.



>   c) A way to say to the layers below it while writing RAM that it
>     needs to go in a given location.

Yes, QEMUFile would need some new APIs accepting offsets, and we
would need a way to report whether a given impl supports random
access, vs streaming only.

>   d) A clean way for a..c only to happen in this case.



>   e) Hmm ram size changes/hotplug/virtio-mem

This is "easy" from libvirt POV, since none of those things are possible
to be invoked during save/restore/migrate.

Hard(er) for QEMU since it doesn't place that restriction on users in the
general case.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|




[Index of Archives]     [Virt Tools]     [Libvirt Users]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]     [Fedora Tools]

  Powered by Linux