-----Original Message-----
From: Xiao Yang <yangx.jy@xxxxxxxxxxxxxx>
Sent: Thursday, January 28, 2021 8:15 PM
To: Leon Romanovsky <leon@xxxxxxxxxx>
Cc: linux-rdma@xxxxxxxxxxxxxxx; jgg@xxxxxxxxxx
Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ
On 2021/1/28 20:54, Leon Romanovsky wrote:
> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
>> On 2021/1/27 20:04, Leon Romanovsky wrote:
>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>>>> module cannot set imm_data in WC when the related WR with imm_data
>>>> finished on SQ.
>>>>
>>>> Signed-off-by: Xiao Yang<yangx.jy@xxxxxxxxxxxxxx>
>>>> ---
>>>>
>>>> Current rxe module and other rdma modules(e.g. mlx5) only set
>>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>>> I am not sure if it is a expected behavior.
>>> This is IBTA behavior.
>>>
>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
>>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>> Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>> RDMA WRITE packets with Immediate Data."
>>>
>>> If I understand the spec, you shouldn't set imm_data in SQ.
>> Hi Leon,
>>
>> About the behavior, I have another question:
>> For send operation with imm_data, we can verify if the delivered
>> imm_data is correct by CQE on RQ.
>> For rdma write operation with imm_data, how to verify if the
>> delivered imm_data is correct? :-)
> Probably that I didn't understand the question, but the RDMA WRITE is
> marked with special opcode in the BTH that indicates imm_data.
Hi Leon,
The quesion is about how to get the imm_data in applications(programs in user space)
1) If client program does send operation with imm_data, server program can get the delivered imm_data by calling ibv_poll_cq(&wc)
2) If client program does rdma write operation with imm_data, server program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
In this case, how does server program get the delivered imm_data?
Best Regards,
Xiao Yang
> Thanks
>
>> Best Regards,
>> Xiao Yang
>>> Thanks
>>>
>>>
>>> .
>>>
>>
>>
>
> .
>
Write with immediate deposits the payload into an RDMA memory region and also consumes a receive work request and generates a receive completion event. The immediate data part sort of looks like a send but without any data in the receive buffer. The WR ID is copied into the completion event along with the immediate data so software can correlate the immediate with the work request.
Bob Pearson
rpearson@xxxxxxx
