-----Original Message----- From: Xiao Yang <yangx.jy@xxxxxxxxxxxxxx> Sent: Thursday, January 28, 2021 11:42 PM To: Bob Pearson <rpearsonhpe@xxxxxxxxx> Cc: Leon Romanovsky <leon@xxxxxxxxxx>; linux-rdma@xxxxxxxxxxxxxxx; jgg@xxxxxxxxxx Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ On 2021/1/28 3:40, Bob Pearson wrote: > On 1/27/21 6:04 AM, Leon Romanovsky wrote: >> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote: >>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe >>> module cannot set imm_data in WC when the related WR with imm_data >>> finished on SQ. >>> >>> Signed-off-by: Xiao Yang<yangx.jy@xxxxxxxxxxxxxx> >>> --- >>> >>> Current rxe module and other rdma modules(e.g. mlx5) only set >>> imm_data in WC when the related WR with imm_data finished on RQ. >>> I am not sure if it is a expected behavior. >> This is IBTA behavior. >> >> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES >> "Immediate Data (ImmDt) contains data that is placed in the receive >> Completion Queue Element (CQE). The ImmDt is only allowed in SEND or >> RDMA WRITE packets with Immediate Data." >> >> If I understand the spec, you shouldn't set imm_data in SQ. >> >> Thanks >> > This seems a little confused to me. wc.imm_data is set in rxe_resp.c in response to an incoming request packet that contains an IMMDT extension header. I.e. a write with immediate or send with immediate opcode from the remote end of the wire. This wc is delivered to the receive completion queue when the message is complete. It should not have anything to do with the local send work queue entries. Hi Bob, Current rdma modules(e.g softroce, mlx5) only set wc_flags to IBV_WC_WITH_IMM for the completed send work queue entries. I am not sure if it is also the IBTA behavior. Best Regards, Xiao Yang > Bob Pearson > > > . > Ah. That might be an issue. I'll check it in the morning and fix it if its wrong. Not sure what the purpose is for the send completion event since the verbs consumer knows what was put in the send work queue entry. On the other hand the receive completion event consumer needs to know whether the imm_data field is valid. Bob
