-----Original Message-----
From: Xiao Yang <yangx.jy@xxxxxxxxxxxxxx>
Sent: Thursday, January 28, 2021 11:42 PM
To: Bob Pearson <rpearsonhpe@xxxxxxxxx>
Cc: Leon Romanovsky <leon@xxxxxxxxxx>; linux-rdma@xxxxxxxxxxxxxxx; jgg@xxxxxxxxxx
Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ
On 2021/1/28 3:40, Bob Pearson wrote:
> On 1/27/21 6:04 AM, Leon Romanovsky wrote:
>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>>> module cannot set imm_data in WC when the related WR with imm_data
>>> finished on SQ.
>>>
>>> Signed-off-by: Xiao Yang<yangx.jy@xxxxxxxxxxxxxx>
>>> ---
>>>
>>> Current rxe module and other rdma modules(e.g. mlx5) only set
>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>> I am not sure if it is a expected behavior.
>> This is IBTA behavior.
>>
>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
>> "Immediate Data (ImmDt) contains data that is placed in the receive
>> Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>> RDMA WRITE packets with Immediate Data."
>>
>> If I understand the spec, you shouldn't set imm_data in SQ.
>>
>> Thanks
>>
> This seems a little confused to me. wc.imm_data is set in rxe_resp.c in response to an incoming request packet that contains an IMMDT extension header. I.e. a write with immediate or send with immediate opcode from the remote end of the wire. This wc is delivered to the receive completion queue when the message is complete. It should not have anything to do with the local send work queue entries.
Hi Bob,
Current rdma modules(e.g softroce, mlx5) only set wc_flags to IBV_WC_WITH_IMM for the completed send work queue entries.
I am not sure if it is also the IBTA behavior.
Best Regards,
Xiao Yang
> Bob Pearson
>
>
> .
>
On inspection, the IBV_EC_WITH_IMM flag is set in rxe_comp.c for send completion queue entries
if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
wqe->wr.opcode == IB_WR_SEND_WITH_IMM)
wc->wc_flags = IB_WC_WITH_IMM;
(There are two copies of this because the user space and kernel space WC structs are different.)
As I said earlier this seems of little value to me because the wc consumer wrote the send wqe and knows
whether it is an immediate operation but doesn't hurt anything. In particular it does not imply that
immediate data is returned in the wc.
It is also set in rxe_resp.c for receive completion entries
if (pkt->mask & RXE_IMMDT_MASK) {
uwc->wc_flags |= IB_WC_WITH_IMM;
uwc->ex.imm_data = immdt_imm(pkt);
}
(Again there are two copies.)
In addition the receive wc opcode is set for write with immediate
wc->opcode = (pkt->mask & RXE_IMMDT_MASK &&
pkt->mask & RXE_WRITE_MASK) ?
IB_WC_RECV_RDMA_WITH_IMM : IB_WC_RECV;
(This is required by IBTA chapter 11.)
The receive wc processing is triggered in the normal case when the last packet of a message is received (i.e. write last with immediate, etc.) which will set the flags above or when an error occurs which may not return the immediate data.
Bob
