Am 2021-01-21 um 2:40 p.m. schrieb Daniel Vetter: > Recently there was a fairly long thread about recoreable hardware page > faults, how they can deadlock, and what to do about that. > > While the discussion is still fresh I figured good time to try and > document the conclusions a bit. Thank you Daniel. This is a good summary of our discussion. It's also an external reference I can point our HW engineers at when they're wondering about what "real software" does. Regards, Felix > > References: https://lore.kernel.org/dri-devel/20210107030127.20393-1-Felix.Kuehling@xxxxxxx/ > Cc: Maarten Lankhorst <maarten.lankhorst@xxxxxxxxxxxxxxx> > Cc: Thomas Hellström <thomas.hellstrom@xxxxxxxxx> > Cc: "Christian König" <christian.koenig@xxxxxxx> > Cc: Jerome Glisse <jglisse@xxxxxxxxxx> > Cc: Felix Kuehling <felix.kuehling@xxxxxxx> > Signed-off-by: Daniel Vetter <daniel.vetter@xxxxxxxxx> > Cc: Sumit Semwal <sumit.semwal@xxxxxxxxxx> > Cc: linux-media@xxxxxxxxxxxxxxx > Cc: linaro-mm-sig@xxxxxxxxxxxxxxxx > -- > I'll be away next week, but figured I'll type this up quickly for some > comments and to check whether I got this all roughly right. > > Critique very much wanted on this, so that we can make sure hw which > can't preempt (with pagefaults pending) like gfx10 has a clear path to > support page faults in upstream. So anything I missed, got wrong or > like that would be good. > -Daniel > --- > Documentation/driver-api/dma-buf.rst | 66 ++++++++++++++++++++++++++++ > 1 file changed, 66 insertions(+) > > diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst > index a2133d69872c..e924c1e4f7a3 100644 > --- a/Documentation/driver-api/dma-buf.rst > +++ b/Documentation/driver-api/dma-buf.rst > @@ -257,3 +257,69 @@ fences in the kernel. This means: > userspace is allowed to use userspace fencing or long running compute > workloads. This also means no implicit fencing for shared buffers in these > cases. > + > +Recoverable Hardware Page Faults Implications > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Modern hardware supports recoverable page faults, which has a lot of > +implications for DMA fences. > + > +First, a pending page fault obviously holds up the work that's running on the > +accelerator and a memory allocation is usually required to resolve the fault. > +But memory allocations are not allowed to gate completion of DMA fences, which > +means any workload using recoverable page faults cannot use DMA fences for > +synchronization. Synchronization fences controlled by userspace must be used > +instead. > + > +On GPUs this poses a problem, because current desktop compositor protocols on > +Linus rely on DMA fences, which means without an entirely new userspace stack > +built on top of userspace fences, they cannot benefit from recoverable page > +faults. The exception is when page faults are only used as migration hints and > +never to on-demand fill a memory request. For now this means recoverable page > +faults on GPUs are limited to pure compute workloads. > + > +Furthermore GPUs usually have shared resources between the 3D rendering and > +compute side, like compute units or command submission engines. If both a 3D > +job with a DMA fence and a compute workload using recoverable page faults are > +pending they could deadlock: > + > +- The 3D workload might need to wait for the compute job to finish and release > + hardware resources first. > + > +- The compute workload might be stuck in a page fault, because the memory > + allocation is waiting for the DMA fence of the 3D workload to complete. > + > +There are a few ways to prevent this problem: > + > +- Compute workloads can always be preempted, even when a page fault is pending > + and not yet repaired. Not all hardware supports this. > + > +- DMA fence workloads and workloads which need page fault handling have > + independent hardware resources to guarantee forward progress. This could be > + achieved through e.g. through dedicated engines and minimal compute unit > + reservations for DMA fence workloads. > + > +- The reservation approach could be further refined by only reserving the > + hardware resources for DMA fence workloads when they are in-flight. This must > + cover the time from when the DMA fence is visible to other threads up to > + moment when fence is completed through dma_fence_signal(). > + > +- As a last resort, if the hardware provides no useful reservation mechanics, > + all workloads must be flushed from the GPU when switching between jobs > + requiring DMA fences or jobs requiring page fault handling: This means all DMA > + fences must complete before a compute job with page fault handling can be > + inserted into the scheduler queue. And vice versa, before a DMA fence can be > + made visible anywhere in the system, all compute workloads must be preempted > + to guarantee all pending GPU page faults are flushed. > + > +Note that workloads that run on independent hardware like copy engines or other > +GPUs do not have any impact. This allows us to keep using DMA fences internally > +in the kernel even for resolving hardware page faults, e.g. by using copy > +engines to clear or copy memory needed to resolve the page fault. > + > +In some ways this page fault problem is a special case of the `Infinite DMA > +Fences` discussions: Infinite fences from compute workloads are allowed to > +depend on DMA fences, but not the other way around. And not even the page fault > +problem is new, because some other CPU thread in userspace might > +hit a page fault which holds up a userspace fence - supporting page faults on > +GPUs doesn't anything fundamentally new.
