On Fri, May 20, 2022 at 5:19 PM Md Haris Iqbal <haris.iqbal@xxxxxxxxx> wrote:
>
> Fixes: 001345339f4c ("RDMA/rxe: Separate HW and SW l/rkeys")
> Signed-off-by: Md Haris Iqbal <haris.iqbal@xxxxxxxxx>
> Signed-off-by: Aleksei Marov <aleksei.marov@xxxxxxxxx>
> ---
> Cc: rpearsonhpe@xxxxxxxxx
> ---
> drivers/infiniband/sw/rxe/rxe_mr.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
> index fc3942e04a1f..04eb804efb23 100644
> --- a/drivers/infiniband/sw/rxe/rxe_mr.c
> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c
> @@ -648,7 +648,7 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe)
>
> mr->access = access;
> mr->lkey = (mr->lkey & ~0xff) | key;
> - mr->rkey = (access & IB_ACCESS_REMOTE) ? mr->lkey : 0;
> + mr->rkey = mr->lkey;
> mr->state = RXE_MR_STATE_VALID;
>
> set = mr->cur_map_set;
> --
> 2.25.1
This code was changed in the below mentioned commit.
commit 001345339f4ca85790a1644a74e33ae77ac116be
Author: Bob Pearson <rpearsonhpe@xxxxxxxxx>
Date: Tue Sep 14 11:42:05 2021 -0500
RDMA/rxe: Separate HW and SW l/rkeys
Separate software and simulated hardware lkeys and rkeys for MRs and MWs.
This makes struct ib_mr and struct ib_mw isolated from hardware changes
triggered by executing work requests.
This change fixes a bug seen in blktest.
Link: https://lore.kernel.org/r/20210914164206.19768-4-rpearsonhpe@xxxxxxxxx
Signed-off-by: Bob Pearson <rpearsonhpe@xxxxxxxxx>
Signed-off-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
After the change, mapping for rnbd/rtrs fails with the following error.
Mapping error,
Client
[Fri May 20 20:27:45 2022] rdma_rxe: qp#17 moved to error state
[Fri May 20 20:27:45 2022] rtrs_server L1211: <blya>: remote access
error (wr_cqe: 000000002d27244a, type: 0, vendor_err: 0x0, len: 0)
Server
[Fri May 20 20:27:45 2022] rnbd_client L597: Mapping device
/dev/nullb0 on session blya, (access_mode: rw, nr_poll_queues: 0)
[Fri May 20 20:27:45 2022] rnbd_client L1208: [session=blya] mapped
4/4 default/read queues.
[Fri May 20 20:27:45 2022] rtrs_client L605: <blya>: RDMA failed:
remote access error
[Fri May 20 20:27:45 2022] rtrs_client L346: <blya>: Failed
IB_WR_LOCAL_INV: WR flushed
[Fri May 20 20:27:45 2022] rtrs_client L605: <blya>: RDMA failed:
remote access error
[Fri May 20 20:27:45 2022] rtrs_client L346: <blya>: Failed
IB_WR_LOCAL_INV: WR flushed
[Fri May 20 20:27:45 2022] rnbd_client L413: </dev/nullb0@blya> read
I/O failed with err: -103
[Fri May 20 20:27:45 2022] I/O error, dev rnbd0, sector 0 op
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Fri May 20 20:27:45 2022] rnbd_client L653: </dev/nullb0@blya> Device
disconnected.
[Fri May 20 20:27:45 2022] Buffer I/O error on dev rnbd0, logical
block 0, async page read
[Fri May 20 20:27:45 2022] rnbd_client L1049: </dev/nullb0@blya> RTRS
failed to transfer IO, err: -103
[Fri May 20 20:27:45 2022] I/O error, dev rnbd0, sector 0 op
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Fri May 20 20:27:45 2022] Buffer I/O error on dev rnbd0, logical
block 0, async page read
[Fri May 20 20:27:45 2022] ldm_validate_partition_table(): Disk read failed.
[Fri May 20 20:27:45 2022] rnbd_client L1049: </dev/nullb0@blya> RTRS
failed to transfer IO, err: -103
[Fri May 20 20:27:45 2022] I/O error, dev rnbd0, sector 0 op
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Fri May 20 20:27:45 2022] Buffer I/O error on dev rnbd0, logical
block 0, async page read
[Fri May 20 20:27:45 2022] rnbd_client L1049: </dev/nullb0@blya> RTRS
failed to transfer IO, err: -103
[Fri May 20 20:27:45 2022] I/O error, dev rnbd0, sector 0 op
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Fri May 20 20:27:45 2022] Buffer I/O error on dev rnbd0, logical
block 0, async page read
[Fri May 20 20:27:45 2022] rnbd0: unable to read partition table
Error on write
Client
[Fri May 20 20:28:39 2022] rdma_rxe: rxe_invalidate_mr: rkey (0x20416)
doesn't match mr->rkey (0x0)
[Fri May 20 20:28:39 2022] rtrs_client L346: <blya>: Failed
IB_WR_LOCAL_INV: local QP operation error
The code for rnbd/rtrs works fine on mlx4 and mlx5.
>From the spec perspective, is there a need for the wr access flag to
be set while doing IB_WR_REG_MR?
>
