On Mon, Aug 13, 2018 at 05:29:31PM +0800, Kenneth Lee wrote: > > I made a quick change basing on the RFCv1 here: > > https://github.com/Kenneth-Lee/linux-kernel-warpdrive/commits/warpdrive-v0.6 > > I just made it compilable and not test it yet. But it shows how the idea is > going to be. > > The Pros is: most of the virtual device stuff can be removed. Resource > management is on the openned files only. > > The Cons is: as Jean said, we have to redo something that has been done by VFIO. > These mainly are: > > 1. Track the dma operation and remove them on resource releasing > 2. Pin the memory with gup and do accounting > > It not going to be easy to make a decision... > Maybe it would be good to list things you want do. Looking at your tree it seems you are re-inventing what dma-buf is already doing. So here is what i understand for SVM/SVA: (1) allow userspace to create a command buffer for a device and bind it to its address space (PASID) (2) allow userspace to directly schedule commands on its command buffer No need to do tracking here as SVM/SVA which rely on PASID and something like PCIE ATS (address translation service). Userspace can shoot itself in the foot but nothing harmful can happen. Non SVM/SVA: (3) allow userspace to wrap a region of its memory into an object so that it can be DMA map (ie GUP + dma_map_page()) (4) Have userspace schedule command referencing object created in (3) using an ioctl. We need to keep track of object usage by the hardware so that we know when it is safe to release resources (dma_unmap_page()). The dma-buf provides everything you want for (3) and (4). With dma-buf you create object and each time it is use by a device you associate a fence with it. When fence is signaled it means that the hardware is done using that object. Fence also allow proper synchronization between multiple devices. For instance making sure that the second device wait for the first device before starting doing its thing. dma-buf documentations is much more thorough explaining all this. Now from implementation point of view, maybe it would be a good idea to create something like the virtual gem driver. It is a virtual device that allow to create GEM object. So maybe we want a virtual device that allow to create dma-buf object from process memory and allow sharing of those dma-buf between multiple devices. Userspace would only have to talk to this virtual device to create object and wrap its memory around, then it could use this object against many actual devices. This decouples the memory management, that can be share between all devices, from the actual device driver, which is obviously specific to every single device. Note that dma-buf use file so that once all file reference are gone the resource can be free and cleanup can happen (dma_unmap_page() ...). This properly handle the resource lifetime issue you seem to worried about. Cheers, Jérôme
