From: Tom Talpey > Sent: 14 April 2021 15:16 > > On 4/12/2021 6:48 PM, Jason Gunthorpe wrote: > > On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote: > > > >> So the issue is only in testing all the providers and platforms, > >> to be sure this new behavior isn't tickling anything that went > >> unnoticed all along, because no RDMA provider ever issued RO. > > > > The mlx5 ethernet driver has run in RO mode for a long time, and it > > operates in basically the same way as RDMA. The issues with Haswell > > have been worked out there already. > > > > The only open question is if the ULPs have errors in their > > implementation, which I don't think we can find out until we apply > > this series and people start running their tests aggressively. > > I agree that the core RO support should go in. But turning it on > by default for a ULP should be the decision of each ULP maintainer. > It's a huge risk to shift all the storage drivers overnight. How > do you propose to ensure the aggressive testing happens? > > One thing that worries me is the patch02 on-by-default for the dma_lkey. > There's no way for a ULP to prevent IB_ACCESS_RELAXED_ORDERING > from being set in __ib_alloc_pd(). What is a ULP in this context? I've presumed that this is all about getting PCIe targets (ie cards) to set the RO (relaxed ordering) bit in some of the write TLP they generate for writing to host memory? So whatever driver initialises the target needs to configure whatever target-specific register enables the RO transfers themselves. After that there could be flags in the PCIe config space of the target and any bridges that clear the RO flag. There could also be flags in the bridges and root complex to ignore the RO flag even if it is set. Then the Linux kernel can have option(s) to tell the driver not to enable RO - even though the driver believes it should all work. This could be a single global flag, or fin-grained in some way. So what exactly is this patch series doing? David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
