Am 07.07.20 um 23:58 schrieb Xiong, Jianxin:
-----Original Message-----
From: Christian König <christian.koenig@xxxxxxx>
Am 03.07.20 um 15:14 schrieb Jason Gunthorpe:
On Fri, Jul 03, 2020 at 02:52:03PM +0200, Daniel Vetter wrote:
So maybe I'm just totally confused about the rdma model. I thought:
- you bind a pile of memory for various transactions, that might
happen whenever. Kernel driver doesn't have much if any insight into
when memory isn't needed anymore. I think in the rdma world that's
called registering memory, but not sure.
Sure, but once registered the memory is able to be used at any moment
with no visibilty from the kernel.
Unlike GPU the transactions that trigger memory access do not go
through the kernel - so there is no ability to interrupt a command
flow and fiddle with mappings.
This is the same for GPUs with user space queues as well.
But we can still say for a process if that this process is using a DMA-buf which is moved out and so can't run any more unless the DMA-buf is
accessible again.
In other words you somehow need to make sure that the hardware is not accessing a piece of memory any more when you want to move it.
While a process can be easily suspended, there is no way to tell the RDMA NIC not to process posted work requests that use specific memory regions (or with any other conditions).
So far it appears to me that DMA-buf dynamic mapping for RDMA is only viable with ODP support. For NICs without ODP, a way to allow pinning the device memory is still needed.
And that's exactly the reason why I introduced explicit pin()/unpin()
functions into the DMA-buf API:
https://elixir.bootlin.com/linux/latest/source/drivers/dma-buf/dma-buf.c#L811
It's just that at least our devices drivers currently prevent P2P with
pinned DMA-buf's for two main reasons:
a) To prevent deny of service attacks because P2P BARs are a rather rare
resource.
b) To prevent failures in configuration where P2P is not always possible
between all devices which want to access a buffer.
Regards,
Christian.
Jianxin
Christian.
Jason
