On 2024-01-24 20:17, Zeng, Oak wrote:
Hi Christian,
Even though I mentioned KFD design, I didn’t mean to copy the KFD
design. I also had hard time to understand the difficulty of KFD under
virtualization environment.
The problem with virtualization is related to virtualization design
choices. There is a single process that proxies requests from multiple
processes in one (or more?) VMs to the GPU driver. That means, we need a
single process with multiple contexts (and address spaces). One proxy
process on the host must support multiple guest address spaces.
I don't know much more than these very high level requirements, and I
only found out about those a few weeks ago. Due to my own bias I can't
comment whether there are bad design choices in the proxy architecture
or in KFD or both. The way we are considering fixing this, is to enable
creating multiple KFD contexts in the same process. Each of those
contexts will still represent a shared virtual address space across
devices (but not the CPU). Because the device address space is not
shared with the CPU, we cannot support our SVM API in this situation.
I still believe that it makes sense to have the kernel mode driver aware
of a shared virtual address space at some level. A per-GPU API and an
API that doesn't require matching CPU and GPU virtual addresses would
enable more flexibility at the cost duplicate information tracking for
multiple devices and duplicate overhead for things like MMU notifiers
and interval tree data structures. Having to coordinate multiple devices
with potentially different address spaces would probably make it more
awkward to implement memory migration. The added flexibility would go
mostly unused, except in some very niche applications.
Regards,
Felix
For us, Xekmd doesn't need to know it is running under bare metal or
virtualized environment. Xekmd is always a guest driver. All the
virtual address used in xekmd is guest virtual address. For SVM, we
require all the VF devices share one single shared address space with
guest CPU program. So all the design works in bare metal environment
can automatically work under virtualized environment. +@Shah, Ankur N
<mailto:ankur.n.shah@xxxxxxxxx> +@Winiarski, Michal
<mailto:michal.winiarski@xxxxxxxxx> to backup me if I am wrong.
Again, shared virtual address space b/t cpu and all gpu devices is a
hard requirement for our system allocator design (which means
malloc’ed memory, cpu stack variables, globals can be directly used in
gpu program. Same requirement as kfd SVM design). This was aligned
with our user space software stack.
For anyone who want to implement system allocator, or SVM, this is a
hard requirement. I started this thread hoping I can leverage the
drm_gpuvm design to manage the shared virtual address space (as the
address range split/merge function was scary to me and I didn’t want
re-invent). I guess my takeaway from this you and Danilo is this
approach is a NAK. Thomas also mentioned to me drm_gpuvm is a overkill
for our svm address range split/merge. So I will make things work
first by manage address range xekmd internally. I can re-look
drm-gpuvm approach in the future.
Maybe a pseudo user program can illustrate our programming model:
Fd0 = open(card0)
Fd1 = open(card1)
Vm0 =xe_vm_create(fd0) //driver create process xe_svm on the process's
first vm_create
Vm1 = xe_vm_create(fd1) //driver re-use xe_svm created above if called
from same process
Queue0 = xe_exec_queue_create(fd0, vm0)
Queue1 = xe_exec_queue_create(fd1, vm1)
//check p2p capability calling L0 API….
ptr = malloc()//this replace bo_create, vm_bind, dma-import/export
Xe_exec(queue0, ptr)//submit gpu job which use ptr, on card0
Xe_exec(queue1, ptr)//submit gpu job which use ptr, on card1
//Gpu page fault handles memory allocation/migration/mapping to gpu
As you can see, from above model, our design is a little bit different
than the KFD design. user need to explicitly create gpuvm (vm0 and vm1
above) for each gpu device. Driver internally have a xe_svm represent
the shared address space b/t cpu and multiple gpu devices. But end
user doesn’t see and no need to create xe_svm. The shared virtual
address space is really managed by linux core mm (through the vma
struct, mm_struct etc). From each gpu device’s perspective, it just
operate under its own gpuvm, not aware of the existence of other
gpuvm, even though in reality all those gpuvm shares a same virtual
address space.
See one more comment inline
*From:*Christian König <christian.koenig@xxxxxxx>
*Sent:* Wednesday, January 24, 2024 3:33 AM
*To:* Zeng, Oak <oak.zeng@xxxxxxxxx>; Danilo Krummrich
<dakr@xxxxxxxxxx>; Dave Airlie <airlied@xxxxxxxxxx>; Daniel Vetter
<daniel@xxxxxxxx>; Felix Kuehling <felix.kuehling@xxxxxxx>
*Cc:* Welty, Brian <brian.welty@xxxxxxxxx>;
dri-devel@xxxxxxxxxxxxxxxxxxxxx; intel-xe@xxxxxxxxxxxxxxxxxxxxx;
Bommu, Krishnaiah <krishnaiah.bommu@xxxxxxxxx>; Ghimiray, Himal Prasad
<himal.prasad.ghimiray@xxxxxxxxx>; Thomas.Hellstrom@xxxxxxxxxxxxxxx;
Vishwanathapura, Niranjana <niranjana.vishwanathapura@xxxxxxxxx>;
Brost, Matthew <matthew.brost@xxxxxxxxx>; Gupta, saurabhg
<saurabhg.gupta@xxxxxxxxx>
*Subject:* Re: Making drm_gpuvm work across gpu devices
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
[SNIP]
Yes most API are per device based.
One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.
Yeah and that was a big mistake in my opinion. We should really not do
that ever again.
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a
virtualization projects. Having SVM as device independent feature
which somehow ties to the process address space turned out to be an
extremely bad idea.
The background is that this only works for some use cases but not all
of them.
What's working much better is to just have a mirror functionality
which says that a range A..B of the process address space is mapped
into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required
for higher level APIs and not something you need at the UAPI or even
inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a
per device basis and *not* tied to the process address space. If you
then get crappy performance because userspace gave contradicting
information where to migrate memory then that's a bug in userspace and
not something the kernel should try to prevent somehow.
[SNIP]
I think if you start using the same drm_gpuvm for multiple devices you
will sooner or later start to run into the same mess we have seen with
KFD, where we moved more and more functionality from the KFD to the DRM
render node because we found that a lot of the stuff simply doesn't work
correctly with a single object to maintain the state.
As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.
Yes, I'm perfectly aware of that. And I can only repeat myself that I
see this design as a rather extreme failure. And I think it's one of
the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending
the CPU process into the GPUs, but this idea only works for a few use
cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with
CPU address != GPU address because the VAs are actually coming from
the guest VM and not the host process.
I don’t get the problem here. For us, under virtualization, both the
cpu address and gpu virtual address operated in xekmd is guest virtual
address. They can still share the same virtual address space (as SVM
required)
Oak
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should
not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to
get explicit permission to do this from Dave and Daniel and maybe even
Linus.
Regards,
Christian.
