On Thu, May 10, 2018 at 04:29:44PM +0200, Christian König wrote: > Am 10.05.2018 um 16:20 schrieb Stephen Bates: > > Hi Jerome > > > > > As it is tie to PASID this is done using IOMMU so looks for caller > > > of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing > > > user is the AMD GPU driver see: > > Ah thanks. This cleared things up for me. A quick search shows there are still no users of intel_svm_bind_mm() but I see the AMD version used in that GPU driver. > > Just FYI: There is also another effort ongoing to give both the AMD, Intel > as well as ARM IOMMUs a common interface so that drivers can use whatever > the platform offers fro SVM support. > > > One thing I could not grok from the code how the GPU driver indicates which DMA events require ATS translations and which do not. I am assuming the driver implements someway of indicating that and its not just a global ON or OFF for all DMAs? The reason I ask is that I looking at if NVMe was to support ATS what would need to be added in the NVMe spec above and beyond what we have in PCI ATS to support efficient use of ATS (for example would we need a flag in the submission queue entries to indicate a particular IO's SGL/PRP should undergo ATS). > > Oh, well that is complicated at best. > > On very old hardware it wasn't a window, but instead you had to use special > commands in your shader which indicated that you want to use an ATS > transaction instead of a normal PCIe transaction for your read/write/atomic. > > As Jerome explained on most hardware we have a window inside the internal > GPU address space which when accessed issues a ATS transaction with a > configurable PASID. > > But on very newer hardware that window became a bit in the GPUVM page > tables, so in theory we now can control it on a 4K granularity basis for the > internal 48bit GPU address space. > To complete this a 50 lines primer on GPU: GPUVA - GPU virtual address GPUPA - GPU physical address GPU run programs very much like CPU program expect a program will have many thousands of threads running concurrently. There is a hierarchy of groups for a given program ie threads are grouped together, the lowest hierarchy level have a group size in <= 64 threads on most GPUs. Those programs (call shader for graphic program think OpenGL, Vulkan or compute for GPGPU think OpenCL CUDA) are submited by the userspace against a given address space. In the "old" days (couple years back when dinausor were still roaming the earth) this address space was specific to the GPU and each user space program could create multiple GPU address space. All the memory operation done by the program was against this address space. Hence all PCIE transactions are spawn from a program + address space. GPU use page table + window aperture (the window aperture is going away so you can focus on page table). To translate GPU virtual address into a physical address. The physical address can point to GPU local memory or to system memory or to another PCIE device memory (ie some PCIE BAR). So all PCIE transaction are spawn through this process of GPUVA to GPUPA then GPUPA is handled by the GPU mmu unit that either spawn a PCIE transaction for non local GPUPA or access local memory otherwise. So per say the kernel driver does not configure which transaction is using ATS or peer to peer. Userspace program create a GPU virtual address space and bind object into it. This object can be system memory or some other PCIE device memory in which case we would to do a peer to peer. So you won't find any logic in the kernel. What you find is creating virtual address space and binding object. Above i talk about the old days, nowadays we want the GPU virtual address space to be exactly the same as the CPU virtual address space as the process which initiate the GPU program is using. This is where we use the PASID and ATS. So here userspace create a special "GPU context" that says that the GPU virtual address space will be the same as the program that create the GPU context. A process ID is then allocated and the mm_struct is bind to this process ID in the IOMMU driver. Then all program executed on the GPU use the process ID to identify the address space against which they are running. All of the above i did not talk about DMA engine which are on the "side" of the GPU to copy memory around. GPU have multiple DMA engines with different capabilities, some of those DMA engine use the same GPU address space as describe above, other use directly GPUPA. Hopes this helps understanding the big picture. I over simplify thing and devils is in the details. Cheers, Jérôme
