Digging out of a hole, apologies to everyone. On Fri, Jun 17, 2022 at 03:08:00PM +0200, Christian König wrote: > Am 17.06.22 um 15:03 schrieb Bas Nieuwenhuizen: > > [SNIP] > > > > > BOOKKEEP can only be used by VM updates themselves. So that they don't > > > > > interfere with CS. > > > > That is the point why we would go BOOKKEEP for explicit sync CS > > > > submissions, no? Explicit submission shouldn't interfere with any > > > > other CS submissions. That includes being totally ignored by GL > > > > importers (if we want to have synchronization there between an > > > > explicit submission and GL, userspace is expected to use Jason's > > > > dmabuf fence import/export IOCTLs) > > > No, that would break existing DMA-buf rules. > > > > > > Explicit CS submissions are still a dependency for implicit submissions. > > This is explicitly what we don't want for explicit submissions and why > > I waited with this series until the DMA_RESV_USAGE series landed. We > > wish to opt out from implicit sync completely, and just use the IOCTLs > > Jason wrote for back-compat with windowing systems that need it. > > > > If BOOKKEEP isn't for that, should we add a new USAGE? > > BOOKKEEP is exactly for that, but as discussed with Daniel that's not what > we want in the kernel. Not sure which Daniel you talked to, but this wasn't me. > When you mix implicit with explicit synchronization (OpenGL with RADV for > example) it should be mandatory for the OpenGL to wait for any RADV > submission before issuing an operation. > > What you want to do is intentionally not supported. vk is very intentional in it's rejecting of any implicit sync. Which means when you share a buffer with gl, even in _that_ case there must be no sync automatically, or your implementation is kinda shit. Instead anyone sharing a buffer with vk and using it in gl must take care of sync by importing the timeline syncobj to gl, that's why all these extensions got added. This leaves libva in the cold, but hey libva didn't even get around to adding the full set of modifier extensions so I can't really get myself to care. So summary this means: - a CS/execbuf for vk should _only_ set BOOKKEEPING fences (except ofc if there's memory management moves in the preparation, which use KERNEL fences and then become additional dependencies for the job) - because vk memory model is that always everything currently bound can be used this means you set BOOKKEEPING on absolutely everything. The current clever trick amdgpu has with shared buffers is also not really the right thing. - implicit sync is only controlled through the new import/export ioctl on the dma-buf - if you set any READ/WRITE fences anywhere else, you have potential oversync compared to what the vk spec would want - userspace gets to keep absolutely all the pieces here. Which is not an issue, because userspace is totally allowed to fill a buffer with garbage and hand that to the compositor already, so there's nothing new going wrong here. - ideally (definitely required for vk sparse) when you unbind or rebind then the BOOKKEEPING fences for the vm/ctx get for the old buffers get simply replaced by the pte clearing and tlb flushing fences (like amdkfd does for compute, vk really just wants to look like compute in everything). In practice, especially with partial and multiple mappings of the same underlying bo involved, this might be too expensive to accurately track since you can only do the replacement trick when the last mapping is gone. It might be worth it for private bo though, dunno. For amdgpu the current special owner checks mostly allow you to get the semantics vulkan wants. But it breaks down when you have cross-device or cross-process sharing. We should probably also document this in the kerneldoc for the BOOKKEEPING usage that this is the fence type that vulkan cs should use in all drivers, otherwise this will become an endless mess of driver specific hacks (i.e. the world we currently live in). -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
