On Mon, Aug 13, 2018 at 10:19:54PM +0000, Howell, Seth wrote: > The user-space and kernel-space enums associated with IB work > request opcodes do not have a one to one correspondence. The first 8 > enum values line up appropriately but the ones after that do > not. Specifically, the opcode IBV_WR_SEND_WITH_INV was being > improperly mapped by the kernel to IB_WR_RDMA_READ_WITH_INV when > communicating with a user-space process. This patch seeks to provide > a mapping of all of the opcodes. Ugh, that is a gross problem to have Also, You need to include a Signed-off-by line in any submissions stating you agree to the Developer Certificate of Origin for your work, see: https://developercertificate.org/ > diff --git a/providers/rxe/rxe.c b/providers/rxe/rxe.c1) > index b9b6c6f9..2c1e3b8b 100644 > +++ b/providers/rxe/rxe.c > @@ -56,6 +56,25 @@ > #include "rxe-abi.h" > #include "rxe.h" > > +/** > + * The userspace ibv_wr_opcode enum does not have a 1:1 correspondence with the kernel's > + * ib_wr_opcode enum. This translation is used to mitigate that difference. > + */ > +static const uint32_t rxe_ib_opcode[] = { > + [IBV_WR_SEND] = IB_WR_SEND, > + [IBV_WR_SEND_WITH_INV] = IB_WR_SEND_WITH_INV, > + [IBV_WR_SEND_WITH_IMM] = IB_WR_SEND_WITH_IMM, > + [IBV_WR_RDMA_WRITE] = IB_WR_RDMA_WRITE, > + [IBV_WR_RDMA_WRITE_WITH_IMM] = IB_WR_RDMA_WRITE_WITH_IMM, > + [IBV_WR_RDMA_READ] = IB_WR_RDMA_READ, > + [IBV_WR_ATOMIC_CMP_AND_SWP] = IB_WR_ATOMIC_CMP_AND_SWP, > + [IBV_WR_ATOMIC_FETCH_AND_ADD] = IB_WR_ATOMIC_FETCH_AND_ADD, > + /* There is no corresponding opcode in the kernel for bind_mw */ > + [IBV_WR_BIND_MW] = IBV_WR_BIND_MW, Ops unsupported by RXE should fail the post send, this should probably map to IB_WR_INVALID ? > + [IBV_WR_LOCAL_INV] = IB_WR_LOCAL_INV, > + [IBV_WR_TSO] = IB_WR_LSO, LSO vs TSO? > +}; > + > static const struct verbs_match_ent hca_table[] = { > /* FIXME: rxe needs a more reliable way to detect the rxe device */ > VERBS_NAME_MATCH("rxe", NULL), > @@ -555,7 +574,7 @@ static void convert_send_wr(struct rxe_send_wr *kwr, struct ibv_send_wr *uwr) > > kwr->wr_id = uwr->wr_id; > kwr->num_sge = uwr->num_sge; > - kwr->opcode = uwr->opcode; > + kwr->opcode = rxe_ib_opcode[uwr->opcode]; > kwr->send_flags = uwr->send_flags; > kwr->ex.imm_data = uwr->imm_data; > > diff --git a/providers/rxe/rxe.h b/providers/rxe/rxe.h > index 96f4ee9c..82ebab75 100644 > +++ b/providers/rxe/rxe.h > @@ -48,6 +48,25 @@ enum rdma_network_type { > RDMA_NETWORK_IPV6 > }; > > +enum { > + IB_WR_RDMA_WRITE =0x00, > + IB_WR_RDMA_WRITE_WITH_IMM =0x01, > + IB_WR_SEND =0x02, > + IB_WR_SEND_WITH_IMM =0x03, > + IB_WR_RDMA_READ =0x04, > + IB_WR_ATOMIC_CMP_AND_SWP =0x05, > + IB_WR_ATOMIC_FETCH_AND_ADD =0x06, > + IB_WR_LSO =0x07, > + IB_WR_SEND_WITH_INV =0x08, > + IB_WR_RDMA_READ_WITH_INV =0x09, > + IB_WR_LOCAL_INV =0x0A, > + IB_WR_REG_MR =0x0B, > + IB_WR_MASKED_ATOMIC_CMP_AND_SWP =0x0C, > + IB_WR_MASKED_ATOMIC_FETCH_AND_ADD =0x0D, > + IB_WR_REG_SIG_MR =0x0E, > + IB_WR_INVALID =0xFF > +}; No.. This needs to be changed in the kernel to move the defines it is using from include/rdma/ib_verbs.h to include/uapi/rdma/ib_user_verbs.h Otherwise they will just get busted up again. Thanks, Jason
