On Fri, Aug 09, 2024 at 03:40:41PM -0500, Michael Galaxy wrote:
Hi Martin, Very good question below.....
Sorry, this thread slipped my mind for a while.
On 8/9/24 06:10, Martin Kletzander wrote:On Wed, Aug 07, 2024 at 10:20:49AM -0500, Michael Galaxy wrote:What we have here so far is the minimal set of changes needed to make it work. I'd like to avoid making this set too complicated. How we handle QMP abstractions can be improved later if we want to engage the original CPR author (steven.sistare@xxxxxxxxxx) at some point.So you are solely relying on the fact that when we start QEMU again it will use the same paths and just resume working? That could be another reason to make changes to libvirt, if only to make sure the paths are the same.Yes, we are relying on that fact, correct. We have not had any serious issues on this front, as when we're doing a live updates, we're expecting all the paths to be the same.Well, if you are starting the domains without libvirt's prior knowledge (i.e. not from as saved snapshot or anything like that) libvirt will generate the paths based on the domain ID. But if there is nothing running during the start libvirt will start the IDs from number 1. Unless the previous VM was also the first one started it will have different ID and the whole path will be different.That is in fact exactly what we are doing. =) I'll be demonstrating this at KVM Forum next month,
That's great, I'm looking forward to it. It might be the best venue to talk about the intricacies of introducing support for such a feature.
but basically, to save time, we ended up starting QEMU *manually* inside the initramfs phase of the boot process. This shaved many valuable seconds off the kexec resume process. We are using libvirt's "domxml-to-native" command to provide us with most of the QEMU command line and then we *manually* bring QEMU back to life before libvirt is back. Later when systemd comes online, libvirt then starts back up again and continues taking ownership of the VM that it started before the kexec occurred. When we do it this way, there are no path naming issues. We did this for many reasons: Pulling libvirt into the initramfs was extremely heavy weight and we did not want to package the whole daemon with all the systemd dependencies inside of the initramfs.... it's just too much baggage. On the other hand, if libvirt had an offline "helper" binary capable of resming a VM inside the initramfs that did not require us bypassing the daemon, that would be really great, but those are thoughts for future work. There are a lot of caveats that we had to workaround in the above approach: the "domxml-to-native" output was not very well sanitized..... it has bugs in it. We can try to post fixes to those fixes when we get around to them. Also, we had to deal with cgroup reconstruction. The cgroups are of course blown away when the new kernel takes over so we had to preserve them across the kexec so that libvirt is able to re-attach to a fully "valid" QEMU when libvirt comes back online. So, there's a lot to unpack there => Much too complex of a conversation for this more minimal patchset.
I see there is much to solve which needs to be worked around it this moment. I understand that this small feature you posted is just the beginning and does not promise to bring the whole CPR feature set to libvirt. I am just trying to be cautious about the future proof aspect of the design. That's because, even though this is a small first step, it might, at least from my experience, bite us in the back later on if there are some details like the need for changing the paths due to something almost irrelevant.
- Michael
Attachment:
signature.asc
Description: PGP signature
