On Wed, Jul 08, 2020 at 11:38:31AM +0200, Christian König wrote: > Am 07.07.20 um 23:58 schrieb Xiong, Jianxin: > > > -----Original Message----- > > > From: Christian König <christian.koenig@xxxxxxx> > > > Am 03.07.20 um 15:14 schrieb Jason Gunthorpe: > > > > On Fri, Jul 03, 2020 at 02:52:03PM +0200, Daniel Vetter wrote: > > > > > > > > > So maybe I'm just totally confused about the rdma model. I thought: > > > > > - you bind a pile of memory for various transactions, that might > > > > > happen whenever. Kernel driver doesn't have much if any insight into > > > > > when memory isn't needed anymore. I think in the rdma world that's > > > > > called registering memory, but not sure. > > > > Sure, but once registered the memory is able to be used at any moment > > > > with no visibilty from the kernel. > > > > > > > > Unlike GPU the transactions that trigger memory access do not go > > > > through the kernel - so there is no ability to interrupt a command > > > > flow and fiddle with mappings. > > > This is the same for GPUs with user space queues as well. > > > > > > But we can still say for a process if that this process is using a DMA-buf which is moved out and so can't run any more unless the DMA-buf is > > > accessible again. > > > > > > In other words you somehow need to make sure that the hardware is not accessing a piece of memory any more when you want to move it. > > > > > While a process can be easily suspended, there is no way to tell the RDMA NIC not to process posted work requests that use specific memory regions (or with any other conditions). > > > > So far it appears to me that DMA-buf dynamic mapping for RDMA is only viable with ODP support. For NICs without ODP, a way to allow pinning the device memory is still needed. > > And that's exactly the reason why I introduced explicit pin()/unpin() > functions into the DMA-buf API: > https://elixir.bootlin.com/linux/latest/source/drivers/dma-buf/dma-buf.c#L811 > > It's just that at least our devices drivers currently prevent P2P with > pinned DMA-buf's for two main reasons: > > a) To prevent deny of service attacks because P2P BARs are a rather rare > resource. > > b) To prevent failures in configuration where P2P is not always possible > between all devices which want to access a buffer. So the above is more or less the question in the cover letter (which didn't make it to dri-devel). Can we somehow throw that limitation out, or is that simply not a good idea? Simply moving buffers to system memory when they're pinned does simplify a lot of headaches. For a specific custom built system we can avoid that maybe, but I think upstream is kinda a different thing. Cheers, Daniel > Regards, > Christian. > > > > > Jianxin > > > > > Christian. > > > > > > > Jason > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
