Shivansh, do you speak for CRIU? Are you able to comment on whether CRIU
supports checkpointing an app that uses uffd?
I do not speak for CRIU - I'm just a user (and hopefully a future contributor), but not a maintainer or owner. I can however comment on whether CRIU supports checkpointing an app that uses UFFD - it doesn't. Looking through both the implementation of CRIU (specifically how they restore memory [1]), and at recently filed Github issues [2], it's pretty clear that CRIU doesn't support processes using UFFD - that they do not currently have plans to [3].
Thanks for all these pointers!
[1] https://github.com/checkpoint-restore/criu/blob/criu-2.x-stable/criu/mem.c#L683
[2] https://github.com/checkpoint-restore/criu/issues/2021
[3] https://github.com/checkpoint-restore/criu/issues/2021#issuecomment-1346971967
Further ... isn't CRIU already using uffd in some cases? ...documentation
mentions [1] that it is used for "lazy (or post-copy) restore in CRIU". At least
if the documentation is correct and its actually implemented.
Shivansh, same question - do you know the current CRIU status/plans for using
uffd-wp instead of soft-dirty? If CRIU doesn't currently implement it and has no
current plans to, how can we guage interest in making a plan?
While I cannot gauge whether the maintainers or main contributors of CRIU plan on using uffd-wp instead of soft-dirty in the future, I can tell you that there is no currently open issue to track that work, and whenever anyone in the past has asked about ARM64 pre-dump support to CRIU (which is the feature that uses soft-dirty/would use uffd-wp), they've always just said it's not supported - but that they do want the feature [4].
So in summary, they want the feature, but no one is working on implementing it (either with soft-dirty or with uffd-wp).
I doubt that CRIU would have any issues with adding the feature via soft-dirty (since, as shown in [4] they're interested in it), but as for using uffd-wp they definitely haven't shown any interest thus far. Based on the fact that it would be a very significant amount of work and it would really only be for ARM64 support (which they're already fine without), I'd be very surprised if they were interested in pursuing it.
Of course, nobody wants to do the work. But that doesn't mean that the
kernel has to do the work :)
If there are some major challenges why it cannot possible be done with
uffd-wp (unfixable), that's a different story.
[4] https://github.com/checkpoint-restore/criu/issues/1859#issuecomment-1972674047
But I'll throw in another idea: do we really need soft-dirty and uffd-wp to
exist at the same time in the same process (or the VMA?). In theory, we
My instinct is that MUXing a PTE bit like this will lead to some subtle problems
that won't appear on arches that support either one or both of the features
independently and unconditionally. Surely better to limit ourselves to either
"arm64 will only support uffd-wp" or "arm64 will support both uffd-wp and
soft-dirty". That way, we could move ahead with reviewing/merging the uffd-wp
support asynchronously to deciding whether we want to support soft-dirty.
My personal preference is having both approaches supported - especially in the context of CRIU since I doubt they'll be willing to rewrite all of the dumping and restore logic just for ARM64 support.
Sure, nobody does any work unless they are forced to.
But this is something that arm64 maintainers will have to decide.
Let's start with uffd-wp that has other well-known users that could
benefit (e.g., QEMU background snapshots).
--
Cheers,
David / dhildenb
