Re: Cross-device and cross-driver HMM support

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Am 03.04.24 um 14:57 schrieb Jason Gunthorpe:
On Wed, Apr 03, 2024 at 11:16:36AM +0200, Christian König wrote:
Am 03.04.24 um 00:57 schrieb Dave Airlie:
On Wed, 27 Mar 2024 at 19:52, Thomas Hellström
<thomas.hellstrom@xxxxxxxxxxxxxxx> wrote:
Hi!

With our SVM mirror work we'll soon start looking at HMM cross-device
support. The identified needs are

1) Instead of migrating foreign device memory to system when the
current device is faulting, leave it in place...
1a) for access using internal interconnect,
1b) for access using PCIE p2p (probably mostly as a reference)
I still agree with Sima that we won't see P2P based on HMM between devices
anytime soon if ever.
We've got a team working on the subset of this problem where we can
have a GPU driver install DEVICE_PRIVATE pages and the RDMA driver use
hmm_range_fault() to take the DEVICE_PRIVATE and return an equivilent
P2P page for DMA.

We already have a working prototype that is not too bad code wise.

The problem with that isn't the software but the hardware.

At least on the AMD GPUs and Intels Xe accelerators we have seen so far page faults are not fast enough to actually work with the semantics the Linux kernel uses for struct pages.

That's why for example the SVM implementation really suck with fork(), the transparent huge page deamon and NUMA migrations.

Somebody should probably sit down and write a performance measurement tool for page faults so that we can start to compare vendors regarding this.

E.g. there is no common representation of DMA addresses with address spaces.
In other words you need to know the device which does DMA for an address to
make sense.
? Every device device calls hmm_range_fault() on it's own, to populate
its own private mirror page table, and gets a P2P page. The device can
DMA map that P2P for its own use to get a topologically appropriate
DMA address for its own private page table. The struct page for P2P
references the pgmap which references the target struct device, the
DMA API provides the requesting struct device. The infrastructure for
all this is all there already.

The problem is the DMA API currently has no idea of inter device connectors like XGMI.

So it can create P2P mappings for PCIe, but anything which isn't part of those interconnects is ignore at the moment as far as I can see.

There is a seperate discussion about optimizing away the P2P pgmap,
but for the moment I'm focused on getting things working by relying on
it.

Additional to that we don't have a representation for internal connections,
e.g. the common kernel has no idea that device A and device B can talk
directly to each other, but not with device C.
We do have this in the PCI P2P framework, it just isn't very complete,
but it does handle the immediate cases I see people building where we
have switches and ACS/!ACS paths with different addressing depending
on topology.

That's not what I meant. I'm talking about direct interconnects which a parallel to the PCIe bus.

As far as I know we haven't even started looking into those.

and we plan to add an infrastructure for this. Probably this can be
done initially without too much (or any) changes to the hmm code
itself.
It is essential any work in this area is not tied to DRM.
hmm_range_fault() and DEVICE_PRIVATE are generic kernel concepts we
need to make them work better not build weird DRM side channels.

Completely agree.

So the question is basically whether anybody is interested in a
drm-wide solution for this and in that case also whether anybody sees
the need for cross-driver support?
We have use cases for this as well, yes.
Unfortunately this is a long journey. The immediate next steps are
Alistair's work to untangle the DAX refcounting mess from ZONE_DEVICE
pages:

https://lore.kernel.org/linux-mm/87ttlhmj9p.fsf@nvdebian.thelocal/

Leon is working on improving the DMA API and RDMA's ODP to
be better setup for this:

https://lore.kernel.org/linux-rdma/cover.1709635535.git.leon@xxxxxxxxxx/

[Which is also the basis for fixing DMABUF's abuse of the DMA API]

Then it is pretty simple to teach hmm_range_fault() to convert a
DEVICE_PRIVATE page into a P2P page using a new pgmap op and from
there the rest already basically exists.

Nice, that's at least one step further than I expected.

Folks doing non-PCIe topologies will need to teach the P2P layer how
address translation works on those buses.

Where to start with that?

Christian.


Jason


[Index of Archives]     [Linux DRI Users]     [Linux Intel Graphics]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux