On Thu, Dec 14, 2017 at 11:53:40AM +0800, Figo.zhang wrote: > 2017-12-14 11:16 GMT+08:00 Jerome Glisse <jglisse@xxxxxxxxxx>: > > > On Thu, Dec 14, 2017 at 10:48:36AM +0800, Figo.zhang wrote: > > > 2017-12-14 0:12 GMT+08:00 Jerome Glisse <jglisse@xxxxxxxxxx>: > > > > > > > On Wed, Dec 13, 2017 at 08:10:42PM +0800, Figo.zhang wrote: [...] > > This is not what happen. Here is the workflow with HMM mirror (note that > > physical address do not matter here so i do not even reference them it is > > all about virtual address): > > 1 They are 3 buffers a, b and r at given virtual address both CPU and > > GPU can access them (concurently or not this does not matter). > > 2 GPU can fault so if any virtual address do not have a page table > > entry inside the GPU page table this trigger a page fault that will > > call HMM mirror helper to snapshot CPU page table into the GPU page > > table. If there is no physical memory backing the virtual address > > (ie CPU page table is also empty for the given virtual address) then > > the regular page fault handler of the kernel is invoked. > > > > so when HMM mirror done, the content of GPU page table entry and > CPU page table entry > are same, right? so the GPU and CPU can access the same physical address, > this physical > address is allocated by CPU malloc systemcall. is it conflict and race > condition? CPU and GPU > write to this physical address concurrently. Correct and yes it is conflict free. PCIE platform already support cache coherent access by device to main memory (snoop transaction in PCIE specification). Access can happen concurently to same byte and it behave exactly the same as if two CPU core try to access the same byte. > > i see this slides said: > http://on-demand.gputechconf.com/gtc/2017/presentation/s7764_john-hubbardgpus-using-hmm-blur-the-lines-between-cpu-and-gpu.pdf > > in page 22~23： > When CPU page fault occurs: > * UM (unified memory driver) copies page data to CPU, umaps from GPU > *HMM maps page to CPU > > when GPU page fault occurs: > *HMM has a malloc record buffer, so UM copy page data to GPU > *HMM unmaps page from CPU > > so in this slides, it said it will has two copies, from CPU to GPU, and > from GPU to CPU. so in this case (mul_mat_on_gpu()), is it really need two > copies in kernel space? This slide is for the case where you use device memory on PCIE platform. When that happen only the device can access the virtual address back by device memory. If CPU try to access such address a page fault is trigger and it migrate the data back to regular memory where both GPU and CPU can access it concurently. And again this behavior only happen if you use HMM non cache coherent device memory model. If you use the device cache coherent model with HMM then CPU can access the device memory directly too and above scenario never happen. Note that memory copy when data move from device to system or from system to device memory are inevitable. This is exactly as with autoNUMA. Also note that in some case thing can get allocated directly on GPU and never copied back to regular memory (only use by GPU and freed once GPU is done with them) the zero copy case. But i want to stress that the zero copy case is unlikely to happen for input buffer. Usualy you do not get your input data set directly on the GPU but from network or disk and you might do pre-processing on CPU (uncompress input, or do something that is better done on the CPU). Then you feed your data to the GPU and you do computation there. > > Without HMM mirror but ATS/PASI (CCIX or CAPI): > > 1 They are 3 buffers a, b and r at given virtual address both CPU and > > GPU can access them (concurently or not this does not matter). > > 2 GPU use the exact same page table as the CPU and fault exactly like > > CPU on empty page table entry > > > > So in the end with HMM mirror or ATS/PASID you get the same behavior. > > There is no complexity like you seem to assume. This all about virtual > > address. At any point in time any given valid virtual address of a process > > point to a given physical memory address and that physical memory address > > is the same on both the CPU and the GPU at any point in time they are > > never out of sync (both in HMM mirror and in ATS/PASID case). > > > > The exception is for platform that do not have CAPI or CCIX property ie > > cache coherency for CPU access to device memory. On such platform when > > you migrate a virtual address to use device physical memory you update > > the CPU page table with a special entry. If the CPU try to access the > > virtual address with special entry it trigger fault and HMM will migrate > > the virtual address back to regular memory. But this does not apply for > > CAPI or CCIX platform. > > > > the example of the virtual address using device physical memory is : gpu_r > = gpu_alloc(m*m*sizeof(float)), > so CPU want to access gpu_r will trigger migrate back to CPU memory, > it will allocate CPU page and copy > to gpu_r's content to CPU pages, right? No. Here we are always talking about virtual address that are the outcome of an mmap syscall either as private anonymous memory or as mmap of regular file (ie not a device file but a regular file on a filesystem). Device driver can migrate any virtual address to use device memory for performance reasons (how, why and when such migration happens is totaly opaque to HMM it is under the control of the device driver). So if you do: BUFA = malloc(size); Then do something with BUFA on the CPU (like reading input or network, ...) the memory is likely to be allocated with regular main memory (like DDR). Now if you start some job on your GPU that access BUFA the device driver might call migrate_vma() helper to migrate the memory to device memory. At that point the virtual address of BUFA point to physical device memory here CAPI or CCIX. If it is not CAPI/CCIX than the GPU page table point to device memory while the CPU page table point to invalid special entry. The GPU can work on BUFA that now reside inside the device memory. Finaly, in the non CAPI/CCIX case, if CPU try to access that memory then a migration back to regular memory happen. What you really need is to decouple the virtual address part from what is the physical memory that is backing a virtual address. HMM provide helpers for both aspect. First to mirror page table so that every virtual address point to same physical address. Second side of HMM is to allow to use device memory transparently inside a process by allowing to migrate any virtual address to use device memory. Both aspect are orthogonal to each others. Cheers, Jérôme -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
