Hi Martin,
Very good question below.....
On 8/9/24 06:10, Martin Kletzander wrote:
On Wed, Aug 07, 2024 at 10:20:49AM -0500, Michael Galaxy wrote:
What we have here so far is the minimal set of changes needed to make it
work.
I'd like to avoid making this set too complicated. How we handle QMP
abstractions
can be improved later if we want to engage the original CPR author
(steven.sistare@xxxxxxxxxx)
at some point.
So you are solely relying on the fact that when we start QEMU again it
will use the same paths and just resume working? That could be another
reason to make changes to libvirt, if only to make sure the paths are
the same.
Yes, we are relying on that fact, correct. We have not had any serious
issues on this front,
as when we're doing a live updates, we're expecting all the paths to be
the same.
Well, if you are starting the domains without libvirt's prior knowledge
(i.e. not from as saved snapshot or anything like that) libvirt will
generate the paths based on the domain ID. But if there is nothing
running during the start libvirt will start the IDs from number 1.
Unless the previous VM was also the first one started it will have
different ID and the whole path will be different.
That is in fact exactly what we are doing. =) I'll be demonstrating this
at KVM Forum next month, but basically, to save time, we ended up
starting QEMU *manually* inside the initramfs phase of the boot process.
This shaved many valuable seconds off the kexec resume process. We are
using libvirt's "domxml-to-native" command to provide us with most of
the QEMU command line and then we *manually* bring QEMU back to life
before libvirt is back. Later when systemd comes online, libvirt then
starts back
up again and continues taking ownership of the VM that it started before
the kexec occurred. When we do it this way, there are no path naming issues.
We did this for many reasons: Pulling libvirt into the initramfs was
extremely
heavy weight and we did not want to package the whole daemon with all
the systemd dependencies inside of the initramfs.... it's just too much
baggage.
On the other hand, if libvirt had an offline "helper" binary capable of
resming a
VM inside the initramfs that did not require us bypassing the daemon,
that would be really
great, but those are thoughts for future work.
There are a lot of caveats that we had to workaround in the above approach:
the "domxml-to-native" output was not very well sanitized..... it has
bugs in it.
We can try to post fixes to those fixes when we get around to them.
Also, we had
to deal with cgroup reconstruction. The cgroups are of course blown away
when
the new kernel takes over so we had to preserve them across the kexec so
that
libvirt is able to re-attach to a fully "valid" QEMU when libvirt comes
back online.
So, there's a lot to unpack there => Much too complex of a conversation
for this
more minimal patchset.
- Michael
