On Sat, Jun 25, 2022 at 1:27 AM Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > On Mon, Jun 13, 2022 at 04:20:36PM +0200, haris iqbal wrote: > > > On 09/06/2022 20:03, Md Haris Iqbal wrote: > > > > Currently rxe_invalidate_mr does invalidate for both local ops, and remote > > > > ones. This means that MR being invalidated is compared with rkey for both, > > > > which is incorrect. For local invalidate, comparison should happen with > > > > lkey, > > > Just checked that IBTA SPEC ”10.6.5“ says that consumer *must* L_Key, R_Key ... > > > Not sure whether we should concern these. > > I agree, 10.6.5 is quite clear that the ULP must present all of the > three options and the HCA can choose any of them. > > So, rxe cannot have a bug if it always uses the rkey ? > > > There are multiple things to consider here.. Since the wr for > > invalidate can have only one invalidate_rkey, there is probably no way > > to send lkey and rkey both as mentioned in the spec. > > In general what this reflects is that in Linux that we don't really > completely support the optional idea in IBA that the HCA can have a > different key space for l and r keys. > > > One way to make this work (mlx does this maybe?) is to always make > > rkey and lkey same and NOT make this dependent on access. Whether an > > MR is open for RDMA operations or not can be checked through the > > access permissions. I am guessing this is how it was working before. > > Yes, no driver in Linux suports a disjoint key space. If disjoint key space is not supported in Linux; does this mean irrespective of whether an MR is registered (IB_WR_REG_MR) for LOCAL or REMOTE access, both rkey and lkey should be set? PS: This discussion started in the following thread, https://marc.info/?l=linux-rdma&m=165390868832428&w=2 > > So, I'm revoking my 'this makes sense to me' - the commit message does > not explain how the IBA requires the use of a lkey for a local > invalidate. > > Jason
