On Fri, Apr 30, 2021 at 09:57:45PM -0400, Felix Kuehling wrote: > We have been working on a prototype supporting CRIU (Checkpoint/Restore > In Userspace) for accelerated compute applications running on AMD GPUs > using ROCm (Radeon Open Compute Platform). We're happy to finally share > this work publicly to solicit feedback and advice. The end-goal is to > get this work included upstream in Linux and CRIU. A short whitepaper > describing our design and intention can be found on Github: > https://github.com/RadeonOpenCompute/criu/tree/criu-dev/test/others/ext-kfd/README.md. > > We have RFC patch series for the kernel (based on Alex Deucher's > amd-staging-drm-next branch) and for CRIU including a new plugin and a > few core CRIU changes. I will send those to the respective mailing lists > separately in a minute. They can also be found on Github. > > CRIU+plugin: https://github.com/RadeonOpenCompute/criu/commits/criu-dev > Kernel (KFD): > https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/commits/fxkamd/criu-wip > > At this point this is very much a work in progress and not ready for > upstream inclusion. There are still several missing features, known > issues, and open questions that we would like to start addressing with > your feedback. Since the thread is a bit split I'm dumping the big thoughts here on this RFC. We've discussed this in the past, but I'm once more (insert meme here) asking whether continuing to walk down the amdgpu vs amdkfd split is really the right choice. It starts to feel a bit much like sunk cost fallacy ... - From the big thread we're having right now on dri-devel it's clear that 3d will also move towards more and more a userspace submit model. But due to backwards compat issues it will be a mixed model, and in some cases we need to pick at runtime which model we're picking. A hard split between the amdgpu and the amdkfd world gets in the way here. - There's use-cases for doing compute in vulkan (that was a discussion from Feb that I kicked again in private, since I think still unresolved). So you need a vulkan stack that runs on both amdgpu and amdvlk. - Maybe not yet on amd's radar, but there's a lot of cloud computing. And maybe they also want CRIU for migrating their containers around. So that means CRIU for amdgpu too, not just amdkf. - What's much worse, and I don't think anyone in amd has realized this yet (at least not in a public thread I've seen). In vulkan you need to be able to switch from compute mode to dma-fence mode after pipelines/devices have been created already. This is because winsys are only initialized in a second step, until that's done you have to pessimistically assume that the user does pure compute. What's worse for buffer sharing you don't even have a clear signal on this stuff. So either - you figure out how to migrate all the buffers and state from amdkfd to amdgpu at runtime, and duplicate all the features. Which is rather pointless. - or you duplicate all the compute features to amdgpu so that vk can use them, and still reasonably easy migrate to winsys/dma-fence mode, which makes amdkfd rather redundant. I've discussed this problem extensively with Jason Ekstrand, and it's really nasty. So purely from a technical pov, only looking at the AMD perspective here, this doesn't make much sense to me. The only reason to keep doubling down on amdkfd I'm seeing is that you've built your compute rocm stack on top of it, and because of that the only option is to keep doing that. Which stops making sense eventually, and we're getting to that point for sure. The other side is a bit the upstream side, but that's a lot smaller: - vulkan compute is one of the more reasonable ways to get cross vendor compute ecosystem off the ground. At least from what I know from background chatter, which you guys probably haven't all heard. amdkfd being the single very odd driver here requiring entirely different uapi for compute mode is not going to be great. - CRIU will need new access rights handling (for the save/restore/resume stuff you're adding). Generally we standardize access rights checks across drivers, and leave everything else to render drivers (command submission, memory management, ...). By adding CRIU support to amdkfd we pretty much guarantee that we wont be able to standardize CRIU access rights across drivers. Which just plains sucks from an upstream/cross-vendor ecosystem pov. And yes we'd still need a per-driver criu plugin in userspace, but the same is true for amdvlk/radv/anv/ and all the other drivers we have: Driver is different, access right management is still the same. And secondly, just because nvidia refuses to collaborate in any standards around gpu compute doesn't mean that's a good reason for us to do the same in upstream. Thirdly, it sounds like this is the first device-driver CRIU support, so I think we need a solid agreement/standard here to set as an example for everyone else. There's all the AI accel chips and fpga-for-compute stuff that I expect will eventually also gain CRIU support. So I'm asking once more, is this _really_ the right path forward? Both for amd (at least long term), but also somewhat for upstream. Cheers, Daniel > What's working and tested at this point: > > * Checkpoint and restore accelerated machine learning apps: PyTorch > running Bert on systems with 1 or 2 GPUs (MI50 or MI100), 100% > unmodified user mode stack > * Checkpoint on one system, restore on a different system > * Checkpoint on one GPU, restore on a different GPU > > Major Known issues: > > * The KFD ioctl API is not final: Needs a complete redesign to allow > future extension without breaking the ABI > * Very slow: Need to implement DMA to dump VRAM contents > > Missing or incomplete features: > > * Support for the new KFD SVM API > * Check device topology during restore > * Checkpoint and restore multiple processes > * Support for applications using Mesa for video decode/encode > * Testing with more different GPUs and workloads > > Big Open questions: > > * What's the preferred way to publish our CRIU plugin? In-tree or > out-of-tree? > * What's the preferred way to distribute our CRIU plugin? Source? > Binary .so? Whole CRIU? Just in-box support? > * If our plugin can be upstreamed in the CRIU tree, what would be the > right directory? > > Best regards, > Felix > > > _______________________________________________ > dri-devel mailing list > dri-devel@xxxxxxxxxxxxxxxxxxxxx > https://lists.freedesktop.org/mailman/listinfo/dri-devel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ amd-gfx mailing list amd-gfx@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/amd-gfx
