On Wed, Jun 29, 2022 at 5:04 PM Pearson, Robert B <robert.pearson2@xxxxxxx> wrote: > > > > -----Original Message----- > From: haris iqbal <haris.phnx@xxxxxxxxx> > Sent: Wednesday, June 29, 2022 7:57 AM > To: Bob Pearson <rpearsonhpe@xxxxxxxxx> > Cc: Jason Gunthorpe <jgg@xxxxxxxx>; lizhijian@xxxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; bvanassche@xxxxxxx; leon@xxxxxxxxxx; haris.iqbal@xxxxxxxxx; jinpu.wang@xxxxxxxxx; aleksei.marov@xxxxxxxxx > Subject: Re: [PATCH] RDMA/rxe: Split rxe_invalidate_mr into local and remote versions > > On Tue, Jun 28, 2022 at 10:04 PM Bob Pearson <rpearsonhpe@xxxxxxxxx> wrote: > > > > On 6/28/22 12:05, haris iqbal wrote: > > > On Tue, Jun 28, 2022 at 6:50 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > >> > > >> On Tue, Jun 28, 2022 at 06:46:39PM +0200, haris iqbal wrote: > > >>> On Tue, Jun 28, 2022 at 6:34 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > >>>> > > >>>> On Tue, Jun 28, 2022 at 06:21:09PM +0200, haris iqbal wrote: > > >>>> > > >>>>>> Yes, no driver in Linux suports a disjoint key space. > > >>>>> > > >>>>> If disjoint key space is not supported in Linux; does this mean > > >>>>> irrespective of whether an MR is registered (IB_WR_REG_MR) for > > >>>>> LOCAL or REMOTE access, both rkey and lkey should be set? > > >>>> > > >>>> No.. It means given any key the driver can always tell if it is a > > >>>> rkey or a lkey. There is never any aliasing of key values. Thus > > >>>> the API often doesn't have a away to tell if the given key is an rkey or lkey. > > >>>> > > >>>>> PS: This discussion started in the following thread, > > >>>>> https://marc.info/?l=linux-rdma&m=165390868832428&w=2 > > >>>> > > >>>> rxe is incorrect to not accept the lkey presented on the > > >>>> invalidate_rkey input. invalidate_rkey is misnamed. > > >>> > > >>> > > >>> Understood. So a better fix for rxe (as compared to the one I > > >>> sent) would be one of the following (if I understand correctly). > > >>> > > >>> 1) The key sent in INV, is compared with lkey or rkey based on > > >>> which one is set (non-zero). > > >> > > >> This one seems to match the spec > > >> > > >> However, it requires that keys don't alias, I don't know if rxe has > > >> done this. > > > > > > Rxe seems to be NOT aliasing for fast reg. Unsure about other cases. > > > > > > Maybe Bob or Zhu can shed some light? > > > > > >> > > >>> OR, > > >>> 2) The key sent in INV, is compared with lkey if the MR has only > > >>> LOCAL access, and rkey if the MR has REMOTE access. > > >> > > >> That might make more sense if rxe has aliasing keys and you need to > > >> be specific about r/l > > >> > > >> Jason > > > > rxe always has lkey=rkey if rkey is asked for else rkey=0 and lkey is always set. > > When you say "rkey is asked for", you mean the access permissions > (LOCAL/REMOTE) asked for that MR right? > > Correct. Thanks. I will send a patch with the discussed changes for invalidate. > > If so, then for RXE to decide which key to compare with while invalidating, it's better to decide based on the access of the MR (basically fix number 2 from my older email). > > > > > Bob
