Re: [bug report] blktests srp/002 hang

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 9/25/23 20:17, Daisuke Matsuda (Fujitsu) wrote:
> On Tue, Sep 26, 2023 12:01 AM Bart Van Assche:
>> On 9/24/23 21:47, Daisuke Matsuda (Fujitsu) wrote:
>>> As Bob wrote above, nobody has found any logical failure in rxe
>>> driver.
>>
>> That's wrong. In case you would not yet have noticed my latest email in
>> this thread, please take a look at
>> https://lore.kernel.org/linux-rdma/e8b76fae-780a-470e-8ec4-c6b650793d10@xxxxxxxxxxxxx/T/#m0fd8ea8a4cbc27b37
>> b042ae4f8e9b024f1871a73.
>> I think the report in that email is a 100% proof that there is a
>> use-after-free issue in the rdma_rxe driver. Use-after-free issues have
>> security implications and also can cause data corruption. I propose to
>> revert the commit that introduced the rdma_rxe use-after-free unless
>> someone comes up with a fix for the rdma_rxe driver.
>>
>> Bart.
> 
> Thank you for the clarification. I see your intention.
> I hope the hang issue will be resolved by addressing this.
> 
> Thanks,
> Daisuke
> 

I have made some progress in understanding the cause of the srp/002 etc. hang.

The two attached files are traces of activity for two qp's qp#151 and qp#167. In my runs of srp/002
All the qp's pass before 167 and all fail after 167 which is the first to fail.

It turns out that all the passing qp's call srp_post_send() some number of times and also call
srp_send_done() the same number of times. Starting at qp#167 the last call to srp_send_done() does
not take place leaving the srp driver waiting for the final completion and causing the hang I believe.

There are four cq's involved in each pair of qp's in the srp test. Two in ib_srp and two in ib_srpt
for the two qp's. Three of them execute completion processing in a soft irq context so the code in
core/cq.c gathers the completions and calls back to the srp drivers. The send side cq in srp uses
cq_direct which requires srp to call ib_process_direct() in order to collect the completions. This
happens in __srp_get_tx_iu() which is called in several places in the srp driver. But only as a side effect
since the purpose of this routine is to get an iu to start a new command.

In the attached files for qp#151 the final call to srp_post_send is followed by the rxe requester and
completer work queues processing the send packet and the ack before a final call to __srp_get_rx_iu()
which gathers the final send side completion and success.

For qp#167 the call to srp_post_send() is followed by the rxe driver processing the send operation and
generating a work completion which is posted to the send cq but there is never a following call to
__srp_get_rx_iu() so the cqe is not received by srp and failure.

I don't yet understand the logic of the srp driver to fix this but the problem is not in the rxe driver
as far as I can tell.

Bob
[  184.877132] qp#151: create_qp
[  184.892362] qp#151: modify_qp: INIT
[  184.892385] qp#151: modify_qp: RTR
[  184.892390] qp#151: modify_qp: RTS
[  184.892722] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  184.893208] ib_srp: qp#151: post-recv:
[  184.893212] ib_srp: qp#151: post-recv:
[  184.893215] ib_srp: qp#151: post-recv:
[  184.893218] ib_srp: qp#151: post-recv:
[  184.893220] ib_srp: qp#151: post-recv:
[  184.893223] ib_srp: qp#151: post-recv:
[  184.893226] ib_srp: qp#151: post-recv:
[  184.893228] ib_srp: qp#151: post-recv:
[  184.893231] ib_srp: qp#151: post-recv:
[  184.893234] ib_srp: qp#151: post-recv:
[  184.893236] ib_srp: qp#151: post-recv:
[  184.893239] ib_srp: qp#151: post-recv:
[  184.893242] ib_srp: qp#151: post-recv:
[  184.893244] ib_srp: qp#151: post-recv:
[  184.893247] ib_srp: qp#151: post-recv:
[  184.893249] ib_srp: qp#151: post-recv:
[  184.893252] ib_srp: qp#151: post-recv:
[  184.893255] ib_srp: qp#151: post-recv:
[  184.893257] ib_srp: qp#151: post-recv:
[  184.893260] ib_srp: qp#151: post-recv:
[  184.893263] ib_srp: qp#151: post-recv:
[  184.893265] ib_srp: qp#151: post-recv:
[  184.893268] ib_srp: qp#151: post-recv:
[  184.893270] ib_srp: qp#151: post-recv:
[  184.893273] ib_srp: qp#151: post-recv:
[  184.893276] ib_srp: qp#151: post-recv:
[  184.893278] ib_srp: qp#151: post-recv:
[  184.893281] ib_srp: qp#151: post-recv:
[  184.893284] ib_srp: qp#151: post-recv:
[  184.893286] ib_srp: qp#151: post-recv:
[  184.893289] ib_srp: qp#151: post-recv:
[  184.893291] ib_srp: qp#151: post-recv:
[  184.893294] ib_srp: qp#151: post-recv:
[  184.893297] ib_srp: qp#151: post-recv:
[  184.893299] ib_srp: qp#151: post-recv:
[  184.893302] ib_srp: qp#151: post-recv:
[  184.893304] ib_srp: qp#151: post-recv:
[  184.893307] ib_srp: qp#151: post-recv:
[  184.893310] ib_srp: qp#151: post-recv:
[  184.893312] ib_srp: qp#151: post-recv:
[  184.893315] ib_srp: qp#151: post-recv:
[  184.893318] ib_srp: qp#151: post-recv:
[  184.893320] ib_srp: qp#151: post-recv:
[  184.893323] ib_srp: qp#151: post-recv:
[  184.893325] ib_srp: qp#151: post-recv:
[  184.893328] ib_srp: qp#151: post-recv:
[  184.893331] ib_srp: qp#151: post-recv:
[  184.893333] ib_srp: qp#151: post-recv:
[  184.893336] ib_srp: qp#151: post-recv:
[  184.893339] ib_srp: qp#151: post-recv:
[  184.893341] ib_srp: qp#151: post-recv:
[  184.893344] ib_srp: qp#151: post-recv:
[  184.893346] ib_srp: qp#151: post-recv:
[  184.893349] ib_srp: qp#151: post-recv:
[  184.893352] ib_srp: qp#151: post-recv:
[  184.893354] ib_srp: qp#151: post-recv:
[  184.893357] ib_srp: qp#151: post-recv:
[  184.893360] ib_srp: qp#151: post-recv:
[  184.893362] ib_srp: qp#151: post-recv:
[  184.893365] ib_srp: qp#151: post-recv:
[  184.893367] ib_srp: qp#151: post-recv:
[  184.893370] ib_srp: qp#151: post-recv:
[  184.893373] ib_srp: qp#151: post-recv:
[  184.893375] ib_srp: qp#151: post-recv:
[  185.127720] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.127760] ib_srp: qp#151: post-reg_mr: 0x207820
[  185.127767] ib_srp: qp#151: post-send:
[  185.127792] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.127805] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.127984] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.127996] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.128182] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.128232] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.128241] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.128254] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.128302] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.128317] ib_srp: qp#151: post-inv_rkey: 0x207820
[  185.128323] ib_srp: qp#151: post-recv:
[  185.128336] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.128388] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.128409] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.128439] ib_srp: qp#151: post-reg_mr: 0x207821
[  185.128446] ib_srp: qp#151: post-send:
[  185.128446] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.128459] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.128548] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.128556] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.128692] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.128736] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.128745] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.128756] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.128769] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.128788] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.128810] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.128821] ib_srp: qp#151: post-reg_mr: 0x20798f
[  185.128821] ib_srp: qp#151: post-inv_rkey: 0x207821
[  185.128828] ib_srp: qp#151: post-send:
[  185.128844] ib_srp: qp#151: post-recv:
[  185.128868] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.128883] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.128891] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.128908] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.128917] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.128921] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.128932] ib_srp: qp#151: post-reg_mr: 0x207822
[  185.128939] ib_srp: qp#151: post-send:
[  185.128946] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.128960] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.129019] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.129026] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.129063] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.129070] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.129285] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.129332] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.129341] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.129352] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.129380] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.129399] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.129404] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.129438] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.129448] ib_srp: qp#151: post-inv_rkey: 0x20798f
[  185.129455] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.129461] ib_srp: qp#151: post-recv:
[  185.129465] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.129471] ib_srp: qp#151: post-inv_rkey: 0x207822
[  185.129475] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.129491] ib_srp: qp#151: post-recv:
[  185.129505] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.129530] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.129535] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.129560] ib_srp: qp#151: post-reg_mr: 0x207823
[  185.129568] ib_srp: qp#151: post-send:
[  185.129570] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.129587] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.129664] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.129672] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.129841] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.129908] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.129917] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.129929] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.129995] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.130010] ib_srp: qp#151: post-inv_rkey: 0x207823
[  185.130034] ib_srp: qp#151: post-recv:
[  185.130035] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.134544] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.134566] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.134593] ib_srp: qp#151: post-reg_mr: 0x207824
[  185.134598] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.134599] ib_srp: qp#151: post-send:
[  185.134618] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.134701] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.134712] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.134845] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.134882] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.134890] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.134898] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.134936] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.134947] ib_srp: qp#151: post-inv_rkey: 0x207824
[  185.134956] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.134961] ib_srp: qp#151: post-recv:
[  185.135496] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.135518] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.135545] ib_srp: qp#151: post-reg_mr: 0x207825
[  185.135552] ib_srp: qp#151: post-send:
[  185.135554] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.135573] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.135678] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.135688] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.135865] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.135913] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.135921] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.135933] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.135994] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.136010] ib_srp: qp#151: post-inv_rkey: 0x207825
[  185.136014] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.136028] ib_srp: qp#151: post-recv:
[  185.141233] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.141260] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.141292] ib_srp: qp#151: post-reg_mr: 0x207826
[  185.141301] ib_srp: qp#151: post-send:
[  185.141302] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.141319] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.141424] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.141428] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.141600] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.141648] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.141663] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.141675] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.141738] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.141754] ib_srp: qp#151: post-inv_rkey: 0x207826
[  185.141762] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.141772] ib_srp: qp#151: post-recv:
[  185.141820] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.141842] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.141882] ib_srp: qp#151: post-reg_mr: 0x207827
[  185.141889] ib_srp: qp#151: post-send:
[  185.141893] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.141909] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.142020] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.142027] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.142199] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.142246] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.142256] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.142271] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.142337] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.142354] ib_srp: qp#151: post-inv_rkey: 0x207827
[  185.142362] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.142370] ib_srp: qp#151: post-recv:
[  185.142463] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.142486] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.142514] ib_srp: qp#151: post-reg_mr: 0x207828
[  185.142523] ib_srp: qp#151: post-send:
[  185.142523] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.142541] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.142657] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.142664] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.142832] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.142880] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.142891] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.142902] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.142962] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.142978] ib_srp: qp#151: post-inv_rkey: 0x207828
[  185.142985] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.142992] ib_srp: qp#151: post-recv:
[  185.143041] ib_srp: qp#151: __srp_get_tx_iu to ib_process_cq_direct
[  185.143062] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  185.143087] ib_srp: qp#151: post-reg_mr: 0x207829
[  185.143093] ib_srp: qp#151: post-send:
[  185.143095] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  185.143111] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  185.143215] enp6s0_rxe: qp#151 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  185.143220] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 0, len: 64
[  185.143393] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  185.143441] enp6s0_rxe: qp#151 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  185.143450] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 128, status: 0, len: 36
[  185.143462] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  185.143530] ib_srp: qp#151: recv-done: opcode: 128 status: 0: len: 36
[  185.143547] ib_srp: qp#151: post-inv_rkey: 0x207829
[  185.143556] enp6s0_rxe: qp#151 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  185.143566] ib_srp: qp#151: post-recv:
[  192.873115] ib_srp: qp#151: srp_destroy_qp to ib_process_cq_direct
[  192.873142] ib_srp: qp#151: send-done: opcode: 0 status: 0: len: 64
[  192.873226] enp6s0_rxe: qp#151 rxe_cq_post: cq#161 opcode: 0, status: 5, len: 0
[  192.973687] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 opcode: 0, status: 5, len: 0
[  192.973701] enp6s0_rxe: qp#151 rxe_cq_post: cq#160 notified
[  195.843870] qp#167: create_qp
[  195.858393] qp#167: modify_qp: INIT
[  195.858402] qp#167: modify_qp: RTR
[  195.858406] qp#167: modify_qp: RTS
[  195.858656] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  195.859199] ib_srp: qp#167: post-recv:
[  195.859203] ib_srp: qp#167: post-recv:
[  195.859205] ib_srp: qp#167: post-recv:
[  195.859208] ib_srp: qp#167: post-recv:
[  195.859210] ib_srp: qp#167: post-recv:
[  195.859213] ib_srp: qp#167: post-recv:
[  195.859216] ib_srp: qp#167: post-recv:
[  195.859218] ib_srp: qp#167: post-recv:
[  195.859221] ib_srp: qp#167: post-recv:
[  195.859224] ib_srp: qp#167: post-recv:
[  195.859226] ib_srp: qp#167: post-recv:
[  195.859229] ib_srp: qp#167: post-recv:
[  195.859231] ib_srp: qp#167: post-recv:
[  195.859234] ib_srp: qp#167: post-recv:
[  195.859237] ib_srp: qp#167: post-recv:
[  195.859239] ib_srp: qp#167: post-recv:
[  195.859242] ib_srp: qp#167: post-recv:
[  195.859244] ib_srp: qp#167: post-recv:
[  195.859247] ib_srp: qp#167: post-recv:
[  195.859250] ib_srp: qp#167: post-recv:
[  195.859253] ib_srp: qp#167: post-recv:
[  195.859255] ib_srp: qp#167: post-recv:
[  195.859258] ib_srp: qp#167: post-recv:
[  195.859260] ib_srp: qp#167: post-recv:
[  195.859263] ib_srp: qp#167: post-recv:
[  195.859266] ib_srp: qp#167: post-recv:
[  195.859268] ib_srp: qp#167: post-recv:
[  195.859271] ib_srp: qp#167: post-recv:
[  195.859274] ib_srp: qp#167: post-recv:
[  195.859276] ib_srp: qp#167: post-recv:
[  195.859279] ib_srp: qp#167: post-recv:
[  195.859281] ib_srp: qp#167: post-recv:
[  195.859284] ib_srp: qp#167: post-recv:
[  195.859287] ib_srp: qp#167: post-recv:
[  195.859289] ib_srp: qp#167: post-recv:
[  195.859292] ib_srp: qp#167: post-recv:
[  195.859294] ib_srp: qp#167: post-recv:
[  195.859297] ib_srp: qp#167: post-recv:
[  195.859300] ib_srp: qp#167: post-recv:
[  195.859303] ib_srp: qp#167: post-recv:
[  195.859306] ib_srp: qp#167: post-recv:
[  195.859308] ib_srp: qp#167: post-recv:
[  195.859311] ib_srp: qp#167: post-recv:
[  195.859313] ib_srp: qp#167: post-recv:
[  195.859316] ib_srp: qp#167: post-recv:
[  195.859319] ib_srp: qp#167: post-recv:
[  195.859321] ib_srp: qp#167: post-recv:
[  195.859324] ib_srp: qp#167: post-recv:
[  195.859326] ib_srp: qp#167: post-recv:
[  195.859329] ib_srp: qp#167: post-recv:
[  195.859332] ib_srp: qp#167: post-recv:
[  195.859334] ib_srp: qp#167: post-recv:
[  195.859337] ib_srp: qp#167: post-recv:
[  195.859339] ib_srp: qp#167: post-recv:
[  195.859342] ib_srp: qp#167: post-recv:
[  195.859345] ib_srp: qp#167: post-recv:
[  195.859348] ib_srp: qp#167: post-recv:
[  195.859350] ib_srp: qp#167: post-recv:
[  195.859353] ib_srp: qp#167: post-recv:
[  195.859356] ib_srp: qp#167: post-recv:
[  195.859358] ib_srp: qp#167: post-recv:
[  195.859361] ib_srp: qp#167: post-recv:
[  195.859364] ib_srp: qp#167: post-recv:
[  195.859366] ib_srp: qp#167: post-recv:
[  196.396284] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[  196.396316] ib_srp: qp#167: post-reg_mr: 0x2458e7
[  196.396325] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  196.396327] ib_srp: qp#167: post-send:
[  196.396338] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  196.396360] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[  196.396383] ib_srp: qp#167: post-reg_mr: 0x245923
[  196.396391] ib_srp: qp#167: post-send:
[  196.396455] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[  196.396478] ib_srp: qp#167: post-reg_mr: 0x245a7d
[  196.396484] ib_srp: qp#167: post-send:
[  196.396590] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[  196.396615] ib_srp: qp#167: post-reg_mr: 0x245b46
[  196.396621] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  196.396624] ib_srp: qp#167: post-send:
[  196.396629] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  196.396645] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  196.396661] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[  196.396662] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  196.396670] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  196.396694] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  196.396702] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_ONLY
[  196.396709] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  196.396733] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  196.396738] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[  196.396746] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[  196.396760] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  196.396770] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[  196.396783] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[  196.396796] ib_srp: qp#167: post-inv_rkey: 0x2458e7
[  196.396798] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.396798] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  196.396804] ib_srp: qp#167: post-recv:
[  196.396813] ib_srp: qp#167: __srp_get_tx_iu to ib_process_cq_direct
[  196.396815] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[  196.396845] ib_srp: qp#167: send-done: opcode: 0 status: 0: len: 64
[  196.396845] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.396850] ib_srp: qp#167: send-done: opcode: 0 status: 0: len: 64
[  196.396855] ib_srp: qp#167: send-done: opcode: 0 status: 0: len: 64
[  196.396860] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  196.396867] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[  196.396886] ib_srp: qp#167: post-reg_mr: 0x2458e8
[  196.396892] ib_srp: qp#167: post-send:
[  196.396898] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.396910] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_REG_MR, length: 0, resid: 0
[  196.396917] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_SEND, length: 64, resid: 64
[  196.397354] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[  196.397373] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[  196.397418] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  196.397429] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[  196.397440] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[  196.397490] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[  196.397513] ib_srp: qp#167: post-inv_rkey: 0x245923
[  196.397534] ib_srp: qp#167: post-recv:
[  196.397576] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[  196.397597] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397615] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397633] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397653] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397668] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[  196.397677] enp6s0_rxe: qp#167 rxe_completer: pkt: opcode = IB_OPCODE_RC_ACKNOWLEDGE
[  196.397685] enp6s0_rxe: qp#167 rxe_cq_post: cq#177 opcode: 0, status: 0, len: 64
[  196.397705] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  196.397714] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[  196.397720] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[  196.397754] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[  196.397771] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397789] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397806] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397827] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397845] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397864] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397883] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397903] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397921] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397925] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.397938] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397957] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397974] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.397991] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398029] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[  196.398031] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[  196.398052] ib_srp: qp#167: post-inv_rkey: 0x245a7d
[  196.398077] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  196.398079] ib_srp: qp#167: post-recv:
[  196.398089] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[  196.398106] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.398113] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[  196.398121] ib_srp: qp#167: post-inv_rkey: 0x245b46
[  196.398122] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.398127] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_FIRST
[  196.398145] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398155] ib_srp: qp#167: post-recv:
[  196.398164] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398185] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398205] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398228] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398245] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398261] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398296] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398334] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398355] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398390] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398411] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398452] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398470] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398505] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398520] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398577] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398594] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398612] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398630] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398676] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398696] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398737] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398756] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398798] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398816] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398853] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398889] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398909] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_MIDDLE
[  196.398947] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_RDMA_WRITE_LAST
[  196.398994] enp6s0_rxe: qp#167 rxe_responder: pkt: opcode = IB_OPCODE_RC_SEND_ONLY
[  196.399001] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 opcode: 128, status: 0, len: 36
[  196.399014] enp6s0_rxe: qp#167 rxe_cq_post: cq#176 notified
[  196.399087] ib_srp: qp#167: recv-done: opcode: 128 status: 0: len: 36
[  196.399107] ib_srp: qp#167: post-inv_rkey: 0x2458e8
[  196.399114] enp6s0_rxe: qp#167 rxe_requester: wqe: IB_WR_LOCAL_INV, length: 0, resid: 0
[  196.399139] ib_srp: qp#167: post-recv:

[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux