Am 14.01.21 um 17:36 schrieb Daniel Vetter:
On Thu, Jan 14, 2021 at 5:01 PM Christian König
<christian.koenig@xxxxxxx> wrote:
Am 14.01.21 um 16:40 schrieb Daniel Vetter:
[SNIP]
So I think we have to somehow solve this in the kernel or we will go in
circles all the time.
So from that pov I think the kernel should at most deal with an
hmm_fence for cross-process communication and maybe some standard wait
primitives (for userspace to use, not for the kernel).
The only use case this would forbid is using page faults for legacy
implicit/explicit dma_fence synced workloads, and I think that's
perfectly ok to not allow. Especially since the motivation here for
all this is compute, and compute doesn't pass around dma_fences
anyway.
As Alex said we will rather soon see this for gfx as well and we most
likely will see combinations of old dma_fence based integrated graphics
with new dedicated GPUs.
So I don't think we can say we reduce the problem to compute and don't
support anything else.
I'm not against pagefaults for gfx, just in pushing the magic into the
kernel. I don't think that works, because it means we add stall points
where usespace, especially vk userspace, really doesn't want it. So
same way like timeline syncobj, we need to push the compat work into
userspace.
There's going to be a few stall points:
- fully new stack, we wait for the userspace fence in the atomic
commit path (which we can, if we're really careful, since we pin all
buffers upfront and so there's no risk)
- userspace fencing gpu in the client, compositor protocol can pass
around userspace fences, but the compositor still uses dma_fence for
itself. There's some stalling in the compositor, which it does already
anyway when it's collecting new frames from clients
- userspace fencing gpu in the client, but no compositor protocol: We
wait in the swapchain, but in a separate thread so that nothing blocks
that shouldn't block
If we instead go with "magic waits in the kernel behind userspace's
back", like what your item 6 would imply, then we're not really
solving anything.
For actual implementation I think the best would be an extension of
drm_syncobj. Those already have at least conceptually future/infinite
fences, and we already have fd passing, so "just" need some protocol
to pass them around. Plus we could use the same uapi for timeline
syncobj using dma_fence as for hmm_fence, so also easier to transition
for userspace to the new world since don't need the new hw capability
to roll out the new uapi and protocols.
That's not that hard to roll out, and technically a lot better than
hacking up dma_resv and hoping we don't end up stalling in wrong
places, which sounds very "eeeek" to me :-)
Yeah, that's what I totally agree upon :)
My idea was just the last resort since we are mixing userspace sync and
memory management so creative here.
Stalling in userspace will probably get some push back as well, but
maybe not as much as stalling in the kernel.
I guess we need to have last-resort stalling in the kernel, but no
more than what we do with drm_syncobj future fences right now. Like
when anything asks for a dma_fence out of an hmm_fence drm_syncob, we
just stall until the hmm_fence is signalled, and then create a
dma_fence that's already signalled and return that to the caller.
Good idea. BTW: We should somehow teach lockdep that this
materialization of any future fence should not happen while holding a
reservation lock?
Obviously this shouldn't happen, since anyone who's timeline aware
will check whether the fence has at least materialized first and stall
somewhere more useful for that first.
Well if I'm not completely mistaken it should help with existing stuff
like an implicit fence for atomic modeset etc...
Ok if we can at least remove implicit sync from the picture then the
question remains how do we integrate HMM into drm_syncobj then?
From an uapi pov probably just an ioctl to create an hmm drm_syncobj,
and a syncobj ioctl to query whether it's a hmm_fence or dma_fence
syncobj, so that userspace can be a bit more clever with where it
should stall - for an hmm_fence the stall will most likely be directly
on the gpu in many cases (so the ioctl should also give us all the
details about that if it's an hmm fence).
I think the real work is going through all the hardware and trying to
figure out what the common ground for userspace fences are. Stuff like
can they be in system memory, or need something special (wc maybe, but
I hope system memory should be fine for everyone), and how you count,
wrap and compare. I also have no idea how/if we can optimized cpu
waits across different drivers.
I think that this is absolutely hardware dependent. E.g. for example AMD
will probably have handles, so that the hardware scheduler can counter
problems like priority inversion.
What we should probably do is to handle this similar to how DMA-buf is
handled - if it's the same driver and device the drm_syncobj we can use
the same handle for both sides.
If it's different driver or device we go through some CPU round trip for
the signaling.
Plus ideally we get some actual wayland protocol going for passing
drm_syncobj around, so we can test it.
And DRI3 :)
Christian.
-Daniel
_______________________________________________
amd-gfx mailing list
amd-gfx@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
