On Wed, Feb 06, 2019 at 01:32:34PM -0500, Dennis Dalessandro wrote:
> Hi folks,
>
> We have a problem with hfi1 that has cropped up after:
>
> 0a93fbcb16e6 ("xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR))"
>
> What this does is overwrite the upper 32 bits of what is the memory address
> (on OPA) so that this gets sent across the wire as the iova. Upon arriving
> on the other side the receiver doesn't like this and tosses it. I'm looking
> into a fix.
How is this possible? iova is basically an identical flow to the
userspace version of this (the virtual base address of the MR)
Is it some kernel specific bug in hfi1?
> Now, a related commit I found for RXE has an interesting comment in the
> commit message:
>
> b024dd0eba6e ("rxe: IB_WR_REG_MR does not capture MR's iova field)"
>
> > FRWR memory registration is done with a series of calls and WRs.
> > 1. ULP invokes ib_dma_map_sg()
> > 2. ULP invokes ib_map_mr_sg()
> > 3. ULP posts an IB_WR_REG_MR on the Send queue
> >
> > Step 2 generates an iova. It is permissible for ULPs to change this
> > iova (with certain restrictions) between steps 2 and 3.
>
> I'm curious what these "certain restrictions" are. Can someone point me to a
> discussion/thread or section of the IBTA spec that explains this?
I think the main restriction is that it still has to match the
alignment requirements of the HCA.
Ie you can't take a transfer starting at physical address 0x1234 and
say the IOVA is actually 0.
Jason
