On 9/25/23 20:17, Daisuke Matsuda (Fujitsu) wrote:
> On Tue, Sep 26, 2023 12:01 AM Bart Van Assche:
>> On 9/24/23 21:47, Daisuke Matsuda (Fujitsu) wrote:
>>> As Bob wrote above, nobody has found any logical failure in rxe
>>> driver.
>>
>> That's wrong. In case you would not yet have noticed my latest email in
>> this thread, please take a look at
>> https://lore.kernel.org/linux-rdma/e8b76fae-780a-470e-8ec4-c6b650793d10@xxxxxxxxxxxxx/T/#m0fd8ea8a4cbc27b37
>> b042ae4f8e9b024f1871a73.
>> I think the report in that email is a 100% proof that there is a
>> use-after-free issue in the rdma_rxe driver. Use-after-free issues have
>> security implications and also can cause data corruption. I propose to
>> revert the commit that introduced the rdma_rxe use-after-free unless
>> someone comes up with a fix for the rdma_rxe driver.
>>
>> Bart.
>
> Thank you for the clarification. I see your intention.
> I hope the hang issue will be resolved by addressing this.
>
> Thanks,
> Daisuke
>
I have made some progress in understanding the cause of the srp/002 etc. hang.
The two attached files are traces of activity for two qp's qp#151 and qp#167. In my runs of srp/002
All the qp's pass before 167 and all fail after 167 which is the first to fail.
It turns out that all the passing qp's call srp_post_send() some number of times and also call
srp_send_done() the same number of times. Starting at qp#167 the last call to srp_send_done() does
not take place leaving the srp driver waiting for the final completion and causing the hang I believe.
There are four cq's involved in each pair of qp's in the srp test. Two in ib_srp and two in ib_srpt
for the two qp's. Three of them execute completion processing in a soft irq context so the code in
core/cq.c gathers the completions and calls back to the srp drivers. The send side cq in srp uses
cq_direct which requires srp to call ib_process_direct() in order to collect the completions. This
happens in __srp_get_tx_iu() which is called in several places in the srp driver. But only as a side effect
since the purpose of this routine is to get an iu to start a new command.
In the attached files for qp#151 the final call to srp_post_send is followed by the rxe requester and
completer work queues processing the send packet and the ack before a final call to __srp_get_rx_iu()
which gathers the final send side completion and success.
For qp#167 the call to srp_post_send() is followed by the rxe driver processing the send operation and
generating a work completion which is posted to the send cq but there is never a following call to
__srp_get_rx_iu() so the cqe is not received by srp and failure.
I don't yet understand the logic of the srp driver to fix this but the problem is not in the rxe driver
as far as I can tell.
Bob
[ 184.877132] qp#151: create_qp
[ 184.892362] qp#151: modify_qp: INIT
[ 184.892385] qp#151: modify_qp: RTR
[ 184.892390] qp#151: modify_qp: RTS
[ 184.892722] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 184.893208] ib_srp: qp#151: post-recv:
[ 184.893212] ib_srp: qp#151: post-recv:
[ 184.893215] ib_srp: qp#151: post-recv:
[ 184.893218] ib_srp: qp#151: post-recv:
[ 184.893220] ib_srp: qp#151: post-recv:
[ 184.893223] ib_srp: qp#151: post-recv:
[ 184.893226] ib_srp: qp#151: post-recv:
[ 184.893228] ib_srp: qp#151: post-recv:
[ 184.893231] ib_srp: qp#151: post-recv:
[ 184.893234] ib_srp: qp#151: post-recv:
[ 184.893236] ib_srp: qp#151: post-recv:
[ 184.893239] ib_srp: qp#151: post-recv:
[ 184.893242] ib_srp: qp#151: post-recv:
[ 184.893244] ib_srp: qp#151: post-recv:
[ 184.893247] ib_srp: qp#151: post-recv:
[ 184.893249] ib_srp: qp#151: post-recv:
[ 184.893252] ib_srp: qp#151: post-recv:
[ 184.893255] ib_srp: qp#151: post-recv:
[ 184.893257] ib_srp: qp#151: post-recv:
[ 184.893260] ib_srp: qp#151: post-recv:
[ 184.893263] ib_srp: qp#151: post-recv:
[ 184.893265] ib_srp: qp#151: post-recv:
[ 184.893268] ib_srp: qp#151: post-recv:
[ 184.893270] ib_srp: qp#151: post-recv:
[ 184.893273] ib_srp: qp#151: post-recv:
[ 184.893276] ib_srp: qp#151: post-recv:
[ 184.893278] ib_srp: qp#151: post-recv:
[ 184.893281] ib_srp: qp#151: post-recv:
[ 184.893284] ib_srp: qp#151: post-recv:
[ 184.893286] ib_srp: qp#151: post-recv:
[ 184.893289] ib_srp: qp#151: post-recv:
[ 184.893291] ib_srp: qp#151: post-recv:
[ 184.893294] ib_srp: qp#151: post-recv:
[ 184.893297] ib_srp: qp#151: post-recv:
[ 184.893299] ib_srp: qp#151: post-recv:
[ 184.893302] ib_srp: qp#151: post-recv:
[ 184.893304] ib_srp: qp#151: post-recv:
[ 184.893307] ib_srp: qp#151: post-recv:
[ 184.893310] ib_srp: qp#151: post-recv:
[ 184.893312] ib_srp: qp#151: post-recv:
[ 184.893315] ib_srp: qp#151: post-recv:
[ 184.893318] ib_srp: qp#151: post-recv:
[ 184.893320] ib_srp: qp#151: post-recv:
[ 184.893323] ib_srp: qp#151: post-recv:
[ 184.893325] ib_srp: qp#151: post-recv:
[ 184.893328] ib_srp: qp#151: post-recv:
[ 184.893331] ib_srp: qp#151: post-recv:
[ 184.893333] ib_srp: qp#151: post-recv:
[ 184.893336] ib_srp: qp#151: post-recv:
[ 184.893339] ib_srp: qp#151: post-recv:
[ 184.893341] ib_srp: qp#151: post-recv:
[ 184.893344] ib_srp: qp#151: post-recv:
[ 184.893346] ib_srp: qp#151: post-recv:
[ 184.893349] ib_srp: qp#151: post-recv:
[ 184.893352] ib_srp: qp#151: post-recv:
[ 184.893354] ib_srp: qp#151: post-recv:
[ 184.893357] ib_srp: qp#151: post-recv:
[ 184.893360] ib_srp: qp#151: post-recv:
[ 184.893362] ib_srp: qp#151: post-recv:
[ 184.893365] ib_srp: qp#151: post-recv:
[ 184.893367] ib_srp: qp#151: post-recv:
[ 184.893370] ib_srp: qp#151: post-recv:
[ 184.893373] ib_srp: qp#151: post-recv:
[ 184.893375] ib_srp: qp#151: post-recv:
[ 185.127720] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.127760] ib_srp: qp#151: post-reg_mr: 0x207820
[ 185.127767] ib_srp: qp#151: post-send:
[ 185.127792] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.127805] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.127984] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.127996] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.128182] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.128232] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.128241] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.128254] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.128302] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.128317] ib_srp: qp#151: post-inv_rkey: 0x207820
[ 185.128323] ib_srp: qp#151: post-recv:
[ 185.128336] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.128388] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.128409] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.128439] ib_srp: qp#151: post-reg_mr: 0x207821
[ 185.128446] ib_srp: qp#151: post-send:
[ 185.128446] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.128459] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.128548] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.128556] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.128692] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.128736] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.128745] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.128756] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.128769] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.128788] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.128810] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.128821] ib_srp: qp#151: post-reg_mr: 0x20798f
[ 185.128821] ib_srp: qp#151: post-inv_rkey: 0x207821
[ 185.128828] ib_srp: qp#151: post-send:
[ 185.128844] ib_srp: qp#151: post-recv:
[ 185.128868] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.128883] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.128891] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.128908] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.128917] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.128921] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.128932] ib_srp: qp#151: post-reg_mr: 0x207822
[ 185.128939] ib_srp: qp#151: post-send:
[ 185.128946] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.128960] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.129019] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.129026] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.129063] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.129070] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.129285] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.129332] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.129341] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.129352] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.129380] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.129399] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.129404] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.129438] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.129448] ib_srp: qp#151: post-inv_rkey: 0x20798f
[ 185.129455] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.129461] ib_srp: qp#151: post-recv:
[ 185.129465] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.129471] ib_srp: qp#151: post-inv_rkey: 0x207822
[ 185.129475] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.129491] ib_srp: qp#151: post-recv:
[ 185.129505] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.129530] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.129535] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.129560] ib_srp: qp#151: post-reg_mr: 0x207823
[ 185.129568] ib_srp: qp#151: post-send:
[ 185.129570] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.129587] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.129664] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.129672] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.129841] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.129908] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.129917] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.129929] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.129995] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.130010] ib_srp: qp#151: post-inv_rkey: 0x207823
[ 185.130034] ib_srp: qp#151: post-recv:
[ 185.130035] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.134544] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.134566] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.134593] ib_srp: qp#151: post-reg_mr: 0x207824
[ 185.134598] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.134599] ib_srp: qp#151: post-send:
[ 185.134618] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.134701] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.134712] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.134845] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.134882] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.134890] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.134898] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.134936] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.134947] ib_srp: qp#151: post-inv_rkey: 0x207824
[ 185.134956] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.134961] ib_srp: qp#151: post-recv:
[ 185.135496] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.135518] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.135545] ib_srp: qp#151: post-reg_mr: 0x207825
[ 185.135552] ib_srp: qp#151: post-send:
[ 185.135554] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.135573] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.135678] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.135688] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.135865] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.135913] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.135921] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.135933] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.135994] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.136010] ib_srp: qp#151: post-inv_rkey: 0x207825
[ 185.136014] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.136028] ib_srp: qp#151: post-recv:
[ 185.141233] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.141260] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.141292] ib_srp: qp#151: post-reg_mr: 0x207826
[ 185.141301] ib_srp: qp#151: post-send:
[ 185.141302] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.141319] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.141424] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.141428] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.141600] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.141648] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.141663] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.141675] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.141738] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.141754] ib_srp: qp#151: post-inv_rkey: 0x207826
[ 185.141762] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.141772] ib_srp: qp#151: post-recv:
[ 185.141820] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.141842] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.141882] ib_srp: qp#151: post-reg_mr: 0x207827
[ 185.141889] ib_srp: qp#151: post-send:
[ 185.141893] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.141909] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.142020] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.142027] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.142199] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.142246] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.142256] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.142271] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.142337] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.142354] ib_srp: qp#151: post-inv_rkey: 0x207827
[ 185.142362] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.142370] ib_srp: qp#151: post-recv:
[ 185.142463] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.142486] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.142514] ib_srp: qp#151: post-reg_mr: 0x207828
[ 185.142523] ib_srp: qp#151: post-send:
[ 185.142523] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.142541] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.142657] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.142664] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.142832] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.142880] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.142891] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.142902] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.142962] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.142978] ib_srp: qp#151: post-inv_rkey: 0x207828
[ 185.142985] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.142992] ib_srp: qp#151: post-recv:
[ 185.143041] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[ 185.143062] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 185.143087] ib_srp: qp#151: post-reg_mr: 0x207829
[ 185.143093] ib_srp: qp#151: post-send:
[ 185.143095] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 185.143111] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 185.143215] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 185.143220] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[ 185.143393] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 185.143441] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 185.143450] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[ 185.143462] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 185.143530] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[ 185.143547] ib_srp: qp#151: post-inv_rkey: 0x207829
[ 185.143556] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 185.143566] ib_srp: qp#151: post-recv:
[ 192.873115] ib_srp: qp#151: srp_destroy_qp to ib_process_cq_direct
[ 192.873142] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[ 192.873226] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 5, len: 0
[ 192.973687] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 0, status: 5, len: 0
[ 192.973701] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[ 195.843870] qp#167: create_qp
[ 195.858393] qp#167: modify_qp: INIT
[ 195.858402] qp#167: modify_qp: RTR
[ 195.858406] qp#167: modify_qp: RTS
[ 195.858656] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 195.859199] ib_srp: qp#167: post-recv:
[ 195.859203] ib_srp: qp#167: post-recv:
[ 195.859205] ib_srp: qp#167: post-recv:
[ 195.859208] ib_srp: qp#167: post-recv:
[ 195.859210] ib_srp: qp#167: post-recv:
[ 195.859213] ib_srp: qp#167: post-recv:
[ 195.859216] ib_srp: qp#167: post-recv:
[ 195.859218] ib_srp: qp#167: post-recv:
[ 195.859221] ib_srp: qp#167: post-recv:
[ 195.859224] ib_srp: qp#167: post-recv:
[ 195.859226] ib_srp: qp#167: post-recv:
[ 195.859229] ib_srp: qp#167: post-recv:
[ 195.859231] ib_srp: qp#167: post-recv:
[ 195.859234] ib_srp: qp#167: post-recv:
[ 195.859237] ib_srp: qp#167: post-recv:
[ 195.859239] ib_srp: qp#167: post-recv:
[ 195.859242] ib_srp: qp#167: post-recv:
[ 195.859244] ib_srp: qp#167: post-recv:
[ 195.859247] ib_srp: qp#167: post-recv:
[ 195.859250] ib_srp: qp#167: post-recv:
[ 195.859253] ib_srp: qp#167: post-recv:
[ 195.859255] ib_srp: qp#167: post-recv:
[ 195.859258] ib_srp: qp#167: post-recv:
[ 195.859260] ib_srp: qp#167: post-recv:
[ 195.859263] ib_srp: qp#167: post-recv:
[ 195.859266] ib_srp: qp#167: post-recv:
[ 195.859268] ib_srp: qp#167: post-recv:
[ 195.859271] ib_srp: qp#167: post-recv:
[ 195.859274] ib_srp: qp#167: post-recv:
[ 195.859276] ib_srp: qp#167: post-recv:
[ 195.859279] ib_srp: qp#167: post-recv:
[ 195.859281] ib_srp: qp#167: post-recv:
[ 195.859284] ib_srp: qp#167: post-recv:
[ 195.859287] ib_srp: qp#167: post-recv:
[ 195.859289] ib_srp: qp#167: post-recv:
[ 195.859292] ib_srp: qp#167: post-recv:
[ 195.859294] ib_srp: qp#167: post-recv:
[ 195.859297] ib_srp: qp#167: post-recv:
[ 195.859300] ib_srp: qp#167: post-recv:
[ 195.859303] ib_srp: qp#167: post-recv:
[ 195.859306] ib_srp: qp#167: post-recv:
[ 195.859308] ib_srp: qp#167: post-recv:
[ 195.859311] ib_srp: qp#167: post-recv:
[ 195.859313] ib_srp: qp#167: post-recv:
[ 195.859316] ib_srp: qp#167: post-recv:
[ 195.859319] ib_srp: qp#167: post-recv:
[ 195.859321] ib_srp: qp#167: post-recv:
[ 195.859324] ib_srp: qp#167: post-recv:
[ 195.859326] ib_srp: qp#167: post-recv:
[ 195.859329] ib_srp: qp#167: post-recv:
[ 195.859332] ib_srp: qp#167: post-recv:
[ 195.859334] ib_srp: qp#167: post-recv:
[ 195.859337] ib_srp: qp#167: post-recv:
[ 195.859339] ib_srp: qp#167: post-recv:
[ 195.859342] ib_srp: qp#167: post-recv:
[ 195.859345] ib_srp: qp#167: post-recv:
[ 195.859348] ib_srp: qp#167: post-recv:
[ 195.859350] ib_srp: qp#167: post-recv:
[ 195.859353] ib_srp: qp#167: post-recv:
[ 195.859356] ib_srp: qp#167: post-recv:
[ 195.859358] ib_srp: qp#167: post-recv:
[ 195.859361] ib_srp: qp#167: post-recv:
[ 195.859364] ib_srp: qp#167: post-recv:
[ 195.859366] ib_srp: qp#167: post-recv:
[ 196.396284] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[ 196.396316] ib_srp: qp#167: post-reg_mr: 0x2458e7
[ 196.396325] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 196.396327] ib_srp: qp#167: post-send:
[ 196.396338] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 196.396360] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[ 196.396383] ib_srp: qp#167: post-reg_mr: 0x245923
[ 196.396391] ib_srp: qp#167: post-send:
[ 196.396455] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[ 196.396478] ib_srp: qp#167: post-reg_mr: 0x245a7d
[ 196.396484] ib_srp: qp#167: post-send:
[ 196.396590] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[ 196.396615] ib_srp: qp#167: post-reg_mr: 0x245b46
[ 196.396621] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 196.396624] ib_srp: qp#167: post-send:
[ 196.396629] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 196.396645] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 196.396661] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[ 196.396662] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 196.396670] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 196.396694] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 196.396702] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[ 196.396709] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 196.396733] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 196.396738] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[ 196.396746] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[ 196.396760] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 196.396770] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[ 196.396783] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[ 196.396796] ib_srp: qp#167: post-inv_rkey: 0x2458e7
[ 196.396798] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.396798] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 196.396804] ib_srp: qp#167: post-recv:
[ 196.396813] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[ 196.396815] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[ 196.396845] ib_srp: qp#167: send-done: opcode: 0 status: 0: len: 64
[ 196.396845] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.396850] ib_srp: qp#167: send-done: opcode: 0 status: 0: len: 64
[ 196.396855] ib_srp: qp#167: send-done: opcode: 0 status: 0: len: 64
[ 196.396860] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 196.396867] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[ 196.396886] ib_srp: qp#167: post-reg_mr: 0x2458e8
[ 196.396892] ib_srp: qp#167: post-send:
[ 196.396898] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.396910] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[ 196.396917] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[ 196.397354] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[ 196.397373] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[ 196.397418] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 196.397429] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[ 196.397440] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[ 196.397490] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[ 196.397513] ib_srp: qp#167: post-inv_rkey: 0x245923
[ 196.397534] ib_srp: qp#167: post-recv:
[ 196.397576] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[ 196.397597] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397615] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397633] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397653] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397668] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[ 196.397677] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[ 196.397685] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[ 196.397705] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 196.397714] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[ 196.397720] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[ 196.397754] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[ 196.397771] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397789] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397806] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397827] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397845] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397864] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397883] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397903] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397921] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397925] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.397938] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397957] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397974] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.397991] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398029] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[ 196.398031] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[ 196.398052] ib_srp: qp#167: post-inv_rkey: 0x245a7d
[ 196.398077] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 196.398079] ib_srp: qp#167: post-recv:
[ 196.398089] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[ 196.398106] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.398113] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[ 196.398121] ib_srp: qp#167: post-inv_rkey: 0x245b46
[ 196.398122] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.398127] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[ 196.398145] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398155] ib_srp: qp#167: post-recv:
[ 196.398164] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398185] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398205] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398228] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398245] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398261] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398296] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398334] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398355] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398390] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398411] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398452] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398470] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398505] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398520] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398577] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398594] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398612] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398630] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398676] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398696] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398737] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398756] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398798] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398816] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398853] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398889] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398909] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[ 196.398947] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[ 196.398994] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[ 196.399001] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[ 196.399014] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[ 196.399087] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[ 196.399107] ib_srp: qp#167: post-inv_rkey: 0x2458e8
[ 196.399114] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[ 196.399139] ib_srp: qp#167: post-recv:
