I haven't seen any replies to this proposal. Either it got lost in the
pre-holiday noise, or there is genuinely no interest in this.
If it's the latter, I would look for an AMDGPU driver-specific solution
with minimally invasive changes in DRM and DMABuf code, if needed. Maybe
it could be generalized later if there is interest then.
Regards,
Felix
On 2023-12-06 16:23, Felix Kuehling wrote:
Executive Summary: We need to add CRIU support to DRM render nodes in
order to maintain CRIU support for ROCm application once they start
relying on render nodes for more GPU memory management. In this email
I'm providing some background why we are doing this, and outlining
some of the problems we need to solve to checkpoint and restore render
node state and shared memory (DMABuf) state. I have some thoughts on
the API design, leaning on what we did for KFD, but would like to get
feedback from the DRI community regarding that API and to what extent
there is interest in making that generic.
We are working on using DRM render nodes for virtual address mappings
in ROCm applications to implement the CUDA11-style VM API and improve
interoperability between graphics and compute. This uses DMABufs for
sharing buffer objects between KFD and multiple render node devices,
as well as between processes. In the long run this also provides a
path to moving all or most memory management from the KFD ioctl API to
libdrm.
Once ROCm user mode starts using render nodes for virtual address
management, that creates a problem for checkpointing and restoring
ROCm applications with CRIU. Currently there is no support for
checkpointing and restoring render node state, other than CPU virtual
address mappings. Support will be needed for checkpointing GEM buffer
objects and handles, their GPU virtual address mappings and memory
sharing relationships between devices and processes.
Eventually, if full CRIU support for graphics applications is desired,
more state would need to be captured, including scheduler contexts and
BO lists. Most of this state is driver-specific.
After some internal discussions we decided to take our design process
public as this potentially touches DRM GEM and DMABuf APIs and may
have implications for other drivers in the future.
One basic question before going into any API details: Is there a
desire to have CRIU support for other DRM drivers?
With that out of the way, some considerations for a possible DRM CRIU
API (either generic of AMDGPU driver specific): The API goes through
several phases during checkpoint and restore:
Checkpoint:
1. Process-info (enumerates objects and sizes so user mode can
allocate memory for the checkpoint, stops execution on the GPU)
2. Checkpoint (store object metadata for BOs, queues, etc.)
3. Unpause (resumes execution after the checkpoint is complete)
Restore:
1. Restore (restore objects, VMAs are not in the right place at this
time)
2. Resume (final fixups after the VMAs are sorted out, resume execution)
For some more background about our implementation in KFD, you can
refer to this whitepaper:
https://github.com/checkpoint-restore/criu/blob/criu-dev/plugins/amdgpu/README.md
Potential objections to a KFD-style CRIU API in DRM render nodes, I'll
address each of them in more detail below:
* Opaque information in the checkpoint data that user mode can't
interpret or do anything with
* A second API for creating objects (e.g. BOs) that is separate from
the regular BO creation API
* Kernel mode would need to be involved in restoring BO sharing
relationships rather than replaying BO creation, export and import
from user mode
# Opaque information in the checkpoint
This comes out of ABI compatibility considerations. Adding any new
objects or attributes to the driver/HW state that needs to be
checkpointed could potentially break the ABI of the CRIU
checkpoint/restore ioctl if the plugin needs to parse that
information. Therefore, much of the information in our KFD CRIU ioctl
API is opaque. It is written by kernel mode in the checkpoint, it is
consumed by kernel mode when restoring the checkpoint, but user mode
doesn't care about the contents or binary layout, so there is no user
mode ABI to break. This is how we were able to maintain CRIU support
when we added the SVM API to KFD without changing the CRIU plugin and
without breaking our ABI.
Opaque information may also lend itself to API abstraction, if this
becomes a generic DRM API with driver-specific callbacks that fill in
HW-specific opaque data.
# Second API for creating objects
Creating BOs and other objects when restoring a checkpoint needs more
information than the usual BO alloc and similar APIs provide. For
example, we need to restore BOs with the same GEM handles so that user
mode can continue using those handles after resuming execution. If BOs
are shared through DMABufs without dynamic attachment, we need to
restore pinned BOs as pinned. Validation of virtual addresses and
handling MMU notifiers must be suspended until the virtual address
space is restored. For user mode queues we need to save and restore a
lot of queue execution state so that execution can resume cleanly.
# Restoring buffer sharing relationships
Different GEM handles in different render nodes and processes can
refer to the same underlying shared memory, either by directly
pointing to the same GEM object, or by creating an import attachment
that may get its SG tables invalidated and updated dynamically through
dynamic attachment callbacks. In the latter case it's obvious, who is
the exporter and who is the importer. In the first case, either one
could be the exporter, and it's not clear who would need to create the
BO and who would need to import it when restoring the checkpoint. To
further complicate things, multiple processes in a checkpoint get
restored concurrently. So there is no guarantee that an exporter has
restored a shared BO at the time an importer is trying to restore its
import.
A proposal to deal with these problems would be to treat importers and
exporters the same. Whoever restores first, ends up creating the BO
and potentially attaching to it. The other process(es) can find BOs
that were already restored by another process by looking it up with a
unique ID that could be based on the DMABuf inode number. An
alternative would be a two-pass approach that needs to restore BOs on
two passes:
1. Restore exported BOs
2. Restore imports
With some inter-process synchronization in CRIU itself between these
two passes. This may require changes in the core CRIU, outside our
plugin. Both approaches depend on identifying BOs with some unique ID
that could be based on the DMABuf inode number in the checkpoint.
However, we would need to identify the processes in the same restore
session, possibly based on parent/child process relationships, to
create a scope where those IDs are valid during restore.
Finally, we would also need to checkpoint and restore DMABuf file
descriptors themselves. These are anonymous file descriptors. The CRIU
plugin could probably be taught to recreate them from the original
exported BO based on the inode number that could be queried with fstat
in the checkpoint. It would need help from the render node CRIU API to
find the right BO from the inode, which may be from a different
process in the same restore session.
Regards,
Felix
