On 27/08/2019 16:37, Jason Gunthorpe wrote: > On Tue, Aug 27, 2019 at 04:22:51PM +0300, Gal Pressman wrote: >> On 27/08/2019 16:17, Jason Gunthorpe wrote: >>> On Tue, Aug 27, 2019 at 03:53:29PM +0300, Gal Pressman wrote: >>>> On 27/08/2019 15:00, Jason Gunthorpe wrote: >>>>> On Tue, Aug 27, 2019 at 11:28:20AM +0300, Gal Pressman wrote: >>>>>> On 26/08/2019 17:05, Gal Pressman wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> Lately I've been seeing DMA-API call traces on our automated testing runs which >>>>>>> complain about overlapping mappings of the same cacheline [1]. >>>>>>> The problem is (most likely) caused due to multiple calls to ibv_reg_mr with the >>>>>>> same address, which as a result DMA maps the same physical addresses more than 7 >>>>>>> (ACTIVE_CACHELINE_MAX_OVERLAP) times. >>>>>> >>>>>> BTW, on rare occasions I'm seeing the boundary check in check_sg_segment [1] >>>>>> fail as well. I don't have a stable repro for it though. >>>>>> >>>>>> Is this a known issue as well? The comment there states it might be a bug in the >>>>>> DMA API implementation, but I'm not sure. >>>>>> >>>>>> [1] https://elixir.bootlin.com/linux/v5.3-rc3/source/kernel/dma/debug.c#L1230 >>>>> >>>>> Maybe we are missing a dma_set_seg_boundary ? >>>>> >>>>> PCI uses low defaults: >>>>> >>>>> dma_set_max_seg_size(&dev->dev, 65536); >>>>> dma_set_seg_boundary(&dev->dev, 0xffffffff); >>>> >>>> What would you set it to? >>> >>> Full 64 bits. >>> >>> For umem the driver is responsible to chop up the SGL as required, not >>> the core code. >> >> But wouldn't this possibly hide driver bugs? Perhaps even in other flows? > > The block stack also uses this information, I've been meaning to check > if we should use dma_attrs in umem so we can have different > parameters. > > I'm not aware of any issue with the 32 bit boundary on RDMA devices.. So something like this? diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 99c4a55545cf..2aa0e48f8dac 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -1199,8 +1199,9 @@ static void setup_dma_device(struct ib_device *device) WARN_ON_ONCE(!parent); device->dma_device = parent; } - /* Setup default max segment size for all IB devices */ + /* Setup default DMA properties for all IB devices */ dma_set_max_seg_size(device->dma_device, SZ_2G); + dma_set_seg_boundary(device->dma_device, U64_MAX); }
