On Thu, Jun 2, 2022 at 7:42 AM Lionel Landwerlin <lionel.g.landwerlin@xxxxxxxxx> wrote: > > On 02/06/2022 00:18, Matthew Brost wrote: > > On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >>> +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an > >>> +async worker. The binding and unbinding will work like a special GPU engine. > >>> +The binding and unbinding operations are serialized and will wait on specified > >>> +input fences before the operation and will signal the output fences upon the > >>> +completion of the operation. Due to serialization, completion of an operation > >>> +will also indicate that all previous operations are also complete. > >> I guess we should avoid saying "will immediately start binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply it's not > >> immediate. > >> > >> > >> I have a question on the behavior of the bind operation when no input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > >> > >> One thing I didn't realize is that because we only get one "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. Note that in Vulkan not every queue has to support sparse binding, so one could consider a dedicated sparse binding only queue family. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > > Question - let's say this done after the above operations: > > > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > > > > Is the exec ordered with respected to bind (i.e. would fence3 & 4 be > > signaled before the exec starts)? > > > > Matt > > > Hi Matt, > > From the vulkan point of view, everything is serialized within an > engine (we map that to a VkQueue). > > So with : > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > EXEC completes first then VM_BIND executes. > > > To be even clearer : > > EXEC (engine=ccs0, in_fence=fence2, out_fence=NULL) > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > > EXEC will wait until fence2 is signaled. > Once fence2 is signaled, EXEC proceeds, finishes and only after it is done, VM_BIND executes. > > It would kind of like having the VM_BIND operation be another batch executed from the ringbuffer buffer. > > -Lionel > > > > > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. I posed the same question on my series for AMD (https://patchwork.freedesktop.org/series/104578/), albeit for slightly different reasons.: if one creates a new VkMemory object, you generally want that mapped ASAP, as you can't track (in a VK_KHR_descriptor_indexing world) whether the next submit is going to use this VkMemory object and hence have to assume the worst (i.e. wait till the map/bind is complete before executing the next submission). If all binds/unbinds (or maps/unmaps) happen in-order that means an operation with input fences could delay stuff we want ASAP. Of course waiting in userspace does have disadvantages: 1) more overhead between fence signalling and the operation, potentially causing slightly bigger GPU bubbles. 2) You can't get an out fence early. Within the driver we can mostly work around this but sync_fd exports, WSI and such will be messy. 3) moving the queue to a thread might make things slightly less ideal due to scheduling delays. Removing the in-order working in the kernel generally seems like madness to me as it is very hard to keep track of the state of the virtual address space (to e.g. track umapping stuff before freeing memory or moving memory around) the one game I tried (FH5 over vkd3d-proton) does sparse mapping as follows: separate queue: 1) 0 cmdbuffer submit with 0 input semaphores and 1 output semaphore 2) sparse bind with input semaphore from 1 and 1 output semaphore 3) 0 cmdbuffer submit with input semaphore from 2 and 1 output fence 4) wait on that fence on the CPU which works very well if we just wait for the sparse bind input semaphore in userspace, but I'm still working on seeing if this is the common usecase or an outlier. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with wait > >> fences in userspace? > >> > >> > >> Sorry I noticed this late. > >> > >> > >> -Lionel > >> > >> >
