On Thu, Jan 17, 2019 at 01:14:06AM -0800, Christoph Hellwig wrote: > On Mon, Jan 14, 2019 at 11:53:08AM -0700, Jason Gunthorpe wrote: > > 'coherent DMA' memory is a property of the cache and when it gets > > remapped to userspace the special cache properties get lost. > > That is not the case. There are a few ways to implement coherent > memory, with the most common ones being: > > a) do nothing (the x86 way) - this will just work from userspace > b) remap the memory with an uncached bit in the PTE. dma_mmap_coherent > will ensure this bit is also set for the userspace mapping. > Everything will just work This I did not know, this driver was just calling io_remap_pfn_range, I think. I'm not sure what it is even doing with this memory, mmaping system pages from kernel to user is kind of backwards from the usual RDMA driver flow.. > > > Also, if the streaming DMA mapping is used, how can the userspace > > > call dma_sync_*? As far as I can tell, the userspace > > > udma_*_device_barrier doesn't seem to replace it. > > > > It can't. platforms that require privileged functionality to do > > dma_sync are not supported by RDMA. Other platforms are required to > > implement their functionality via barriers. > > While there might be a few architectures were cache invalidation / > writeback is privileged this is not the common case as far as I can > tell. In general those are available to userspace code as well. Look at ARM. It has 16 different implementations for cache flushing (see arch/arm/mm/cache-*). Many of those require privilege. The ARM11 L1 cache docs say that it's flush ops require privilege, see: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0360f/BABJCGJG.html All CP15 c7 operations can only be performed in a privileged mode, except Data Synchronization Barrier, Flush Prefetch Buffer, and Data Memory Barrier. These can be performed in User and privileged modes. Attempting to perform a privileged operation in User mode results in an Undefined Instruction exception. If you look at how the ARM 'outer_cache' ops work in the kernel, notice all the arguments require PA's not VAs. Obviously we can't have userspace running around cache flushing arbitary PA's so this is all privileged stuff too. There may be some exceptions, on certain chips, but the general direction is to require privilege here. Even if we could execute the CPU instructions there is no reliable way for userspace to know which combination of implementations it needs to use, and what parameters the cache is running with. Most likely we'd have to approach this via the VDSO or something.. bleck MIPS is broadly similar from my memory. No idea on the others, but if ARM and MIPS require priv for their L2's then I bet most other CPUs follow.. So, you really need a syscall to implement this stuff.. which is why nobody cares - it would be too performance sucking to make any sense.. Especially for something like EFA which is only deployed by AWS and only on DMA coherent platforms. Jason
