2017-12-14 11:16 GMT+08:00 Jerome Glisse <jglisse@xxxxxxxxxx>:
On Thu, Dec 14, 2017 at 10:48:36AM +0800, Figo.zhang wrote:
> 2017-12-14 0:12 GMT+08:00 Jerome Glisse <jglisse@xxxxxxxxxx>:
>
> > On Wed, Dec 13, 2017 at 08:10:42PM +0800, Figo.zhang wrote:
[...]
> > Basic example is without HMM:
> > mul_mat_on_gpu(float *r, float *a, float *b, unsigned m)
> > {
> > gpu_buffer_t gpu_r, gpu_a, gpu_b;
> >
> > gpu_r = gpu_alloc(m*m*sizeof(float));
> > gpu_a = gpu_alloc(m*m*sizeof(float));
> > gpu_b = gpu_alloc(m*m*sizeof(float));
> > gpu_copy_to(gpu_a, a, m*m*sizeof(float));
> > gpu_copy_to(gpu_b, b, m*m*sizeof(float));
> >
> > gpu_mul_mat(gpu_r, gpu_a, gpu_b, m);
> >
> > gpu_copy_from(gpu_r, r, m*m*sizeof(float));
> > }
> >
>
> The traditional workflow is:
> 1. the pointer a, b and r are total point to the CPU memory
> 2. create/alloc three GPU buffers: gpu_a, gpu_b, gpu_r
> 3. copy CPU memory a and b to GPU memory gpu_b and gpu_b
> 4. let the GPU to do the calculation
> 5. copy the result from GPU buffer (gpu_r) to CPU buffer (r)
>
> is it right?
Right.
This is not what happen. Here is the workflow with HMM mirror (note that
> > With HMM:
> > mul_mat_on_gpu(float *r, float *a, float *b, unsigned m)
> > {
> > gpu_mul_mat(r, a, b, m);
> > }
> >
>
> with HMM workflow:
> 1. CPU has three buffer: a, b, r, and it is physical addr is : pa, pb, pr
> and GPU has tree physical buffer: gpu_a, gpu_b, gpu_r
> 2. GPU want to access buffer a and b, cause a GPU page fault
> 3. GPU report a page fault to CPU
> 4. CPU got a GPU page fault:
> * unmap the buffer a,b,r (who do it? GPU driver?)
> * copy the buffer a ,b's content to GPU physical buffers:
> gpu_a, gpu_b
> * fill the GPU page table entry with these pages (gpu_a,
> gpu_b, gpu_r) of the CPU virtual address: a,b,r;
>
> 5. GPU do the calculation
> 6. CPU want to get result from buffer r and will cause a CPU page fault:
> 7. in CPU page fault:
> * unmap the GPU page table entry for virtual address a,b,r.
> (who do the unmap? GPU driver?)
> * copy the GPU's buffer content (gpu_a, gpu_b, gpu_r) to
> CPU buffer (abr)
> * fill the CPU page table entry: virtual_addr -> buffer
> (pa,pb,pr)
> 8. so the CPU can get the result form buffer r.
>
> my guess workflow is right?
> it seems need two copy, from CPU to GPU, and then GPU to CPU for result.
> * is it CPU and GPU have the page table concurrently, so
> no page fault occur?
> * how about the performance? it sounds will create lots of page fault.
physical address do not matter here so i do not even reference them it is
all about virtual address):
1 They are 3 buffers a, b and r at given virtual address both CPU and
GPU can access them (concurently or not this does not matter).
2 GPU can fault so if any virtual address do not have a page table
entry inside the GPU page table this trigger a page fault that will
call HMM mirror helper to snapshot CPU page table into the GPU page
table. If there is no physical memory backing the virtual address
(ie CPU page table is also empty for the given virtual address) then
the regular page fault handler of the kernel is invoked.
so when HMM mirror done, the content of GPU page table entry and CPU page table entry
are same, right? so the GPU and CPU can access the same physical address, this physical
address is allocated by CPU malloc systemcall. is it conflict and race condition? CPU and GPU
write to this physical address concurrently.
i see this slides said:
in page 22~23：
When CPU page fault occurs:
* UM (unified memory driver) copies page data to CPU, umaps from GPU
*HMM maps page to CPU
when GPU page fault occurs:
*HMM has a malloc record buffer, so UM copy page data to GPU
*HMM unmaps page from CPU
so in this slides, it said it will has two copies, from CPU to GPU, and
from GPU to CPU. so in this case (mul_mat_on_gpu()), is it really need two copies in kernel space?
Without HMM mirror but ATS/PASI (CCIX or CAPI):
1 They are 3 buffers a, b and r at given virtual address both CPU and
GPU can access them (concurently or not this does not matter).
2 GPU use the exact same page table as the CPU and fault exactly like
CPU on empty page table entry
So in the end with HMM mirror or ATS/PASID you get the same behavior.
There is no complexity like you seem to assume. This all about virtual
address. At any point in time any given valid virtual address of a process
point to a given physical memory address and that physical memory address
is the same on both the CPU and the GPU at any point in time they are
never out of sync (both in HMM mirror and in ATS/PASID case).
The exception is for platform that do not have CAPI or CCIX property ie
cache coherency for CPU access to device memory. On such platform when
you migrate a virtual address to use device physical memory you update
the CPU page table with a special entry. If the CPU try to access the
virtual address with special entry it trigger fault and HMM will migrate
the virtual address back to regular memory. But this does not apply for
CAPI or CCIX platform.
the example of the virtual address using device physical memory is : gpu_r = gpu_alloc(m*m*sizeof(float)),
so CPU want to access gpu_r will trigger migrate back to CPU memory, it will allocate CPU page and copy
to gpu_r's content to CPU pages, right?
Too minimize page fault the device driver is encourage to pre-fault and
prepopulate its page table (the HMM mirror case). Often device driver has
enough context information to guess what range of virtual address is
about to be access by the device and thus pre-fault thing.
Hope this clarify thing for you.
Cheers,
Jérôme
