Jason,
Looking at the current tree post 5.10.0-rc1 rxe is pretty broken.
The first problem I have run into is caused by the following recent patch:
commit f959dcd6ddfd29235030e8026471ac1b022ad2b0
Author: Thomas Tai <thomas.tai@xxxxxxxxxx>
Date: Thu Sep 17 18:43:03 2020 +0200
dma-direct: Fix potential NULL pointer dereference
When booting the kernel v5.9-rc4 on a VM, the kernel would panic when
printing a warning message in swiotlb_map(). The dev->dma_mask must not
be a NULL pointer when calling the dma mapping layer. A NULL pointer
check can potentially avoid the panic.
Signed-off-by: Thomas Tai <thomas.tai@xxxxxxxxxx>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Signed-off-by: Christoph Hellwig <hch@xxxxxx>
which causes a warning if ib_device->dma_device->dma_mask is NULL which it is at the moment for rxe.
But, rxe never does any DMA. It only copies data in software. It does however sometimes do this copy at interrupt level and will fail badly if the users memory is paged out. Rxe just follows the HW rdma model and pins MRs which solves this problem. The original rxe code does packet processing in tasklets which are software interrupts or on the NAPI thread which is also an interrupt.
Rxe calls ib_register_device() with a NULL 3rd argument which is supported. But in this case, ib_register_device just sets dma_device to point at &ib_device->dev which is sort of a dummy device and never sets dma_mask to anything. This now fails because of the above patch.
I can think of three approaches to fixing this problem:
1. Fix ib_register_device to correctly handle a NULL dma_device argument (if anyone besides rxe uses it)
2. Fix rxe to provide a valid dma_device. Currently rxe assumes that ib_device->dev will be it and sets dev.parent to point at the dma_device behind the network device that has rxe installed on it.
3. (Much harder) Strip out all the memory pinning and DMA support and perform the copies in work queues which can sleep. The problem is that rdma code basically assumes that DMA is always there.
This may be easier than I am seeing. Let me know what you suggest.
Thanks,
Bob
