> On Thu, Jul 14, 2022 at 02:20:06PM +0000, Tung, Chien Tin wrote:
> > No one is interested in helping with Create QP issue with Mellanox
> controller?
>
> 1. Both Jason and I had personal issues that prevented from us to
> participate in linux-rdma ML.
Okay,, I don't know what that means but I won't dwell on it.
> 2. max_qp is set by FW. Please contact NVidia support.
I sent this issue to the mailing list because with different kernel I get different
number with 5.18.5 creating 1/2 of QPs as RHEL 8.5/MOFED.
If you still think this is a FW issue, I can certainly take it up with support.
Thanks,
Chien
> > > All tests are performed using the same server, 64G memory, RHEL 8.5,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> same
> > > ulimit setting and selinux disabled.
> > > ConnectX-4 has firmware-version: 12.28.2006 (MT_2180110032), in
> infiniband
> > > mode and max_qp: 262144.
> > >
> > > Test 1 SW config - RHEL 8.5, MLNX_OFED_LINUX-5.5-1.0.3.2:, max qp
> created
> > > - 261935
> > > Test 2 SW config - same as test 1 with MOFED except rdma-core library is
> > > upstream c590cbd51632, max qp created - 261935 (same as test 1)
> > > Test 3 SW config - MOFED uninstalled, 5.18.5 kernel installed, rdma-core
> > > library is upstream c590cbd51632, max qp created 130863
> > >
> > > Can someone help me understand why Test 3 is half of the number? Also,
> is it
> > > possible to hit the max_qp number of 262144?
> > > Not to muddle the issue too much, I have observed similar behavior with
> > > ConnectX-6 (130763 and 261875 QPs) but don't have logs with those runs.
> > >
> > > Thanks,
> > >
> > > Chien
> > >
> > > I've interlaced "rdma resource" prints in the dmesg log by pausing test
> > > program in gdb with breakpoint set to ibv_create_qp.
> > >
> > > <Test 1 log>
> > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261933 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 920.711591] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 920.711597] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 920.711601] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 920.711604] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 920.711606] cmd[0]: 020: 00384800 00000015 0003ff5d 04000000
> > > [ 920.711608] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 920.711609] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 920.711611] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 920.711613] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 920.711614] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 920.711616] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 920.711617] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 920.711619] cmd[0]: 0a0: 00000000 00000000 00000300 00000000
> > > [ 920.711621] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe7c0
> > > [ 920.711622] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 920.711624] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 920.711625] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 920.711627] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 920.711629] cmd[0]: 100: 00000001 00000000 00000000 00000000
> > > [ 920.711630] cmd[0]: 110: 00000006 180c0000 00000005 44200000
> > >
> > > [ 920.711634] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 920.712125] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 920.712129] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 920.712131] cmd[0]: 000: 00000000 00000000 00040121 00000000
> > >
> > > [ 920.712133] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > >
> > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261934 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 938.006403] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 938.006408] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 938.006411] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 938.006414] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 938.006416] cmd[0]: 020: 00384800 00000015 0003ff5e 00000000
> > > [ 938.006418] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 938.006419] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 938.006421] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 938.006422] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 938.006424] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 938.006425] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 938.006427] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 938.006429] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [ 938.006431] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe800
> > > [ 938.006433] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 938.006434] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 938.006436] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 938.006438] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 938.006439] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [ 938.006441] cmd[0]: 110: 00000003 5ed0e000 00000003 5ed0f000
> > > [ 938.006443] cmd[0]: 120: 00000006 1820a000 00000002 f1f87000
> > > [ 938.006444] cmd[0]: 130: 00000002 f1fcf000 00000006 1805f000
> > > [ 938.006446] cmd[0]: 140: 00000003 8efe1000 00000006 1805d000
> > > [ 938.006448] cmd[0]: 150: 00000004 956b5000 00000000 00000000
> > >
> > > [ 938.006451] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 938.006859] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 938.006864] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 938.006866] cmd[0]: 000: 00000000 00000000 00040122 00000000
> > >
> > > [ 938.006869] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > >
> > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 952.880460] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 952.880465] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 952.880468] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 952.880471] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 952.880473] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000
> > > [ 952.880475] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 952.880476] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 952.880478] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 952.880480] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 952.880481] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 952.880483] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 952.880485] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 952.880486] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [ 952.880488] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe840
> > > [ 952.880490] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 952.880491] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 952.880493] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 952.880494] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 952.880496] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [ 952.880498] cmd[0]: 110: 00000001 d3ae6000 00000002 f1ef7000
> > > [ 952.880499] cmd[0]: 120: 00000003 ae09f000 00000001 28268000
> > > [ 952.880501] cmd[0]: 130: 00000003 ae007000 00000001 b4cf4000
> > > [ 952.880503] cmd[0]: 140: 00000006 180fc000 00000002 2a9e6000
> > > [ 952.880504] cmd[0]: 150: 00000002 f1ed0000 00000002 88295000
> > >
> > > [ 952.880508] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 952.881353] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 952.881358] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 952.881360] cmd[0]: 000: 05000000 0065b500 00000000 00000000
> > >
> > > [ 952.881363] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > > [ 952.881401] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 4323):
> > > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5),
> syndrome
> > > (0x65b500)
> > > [ 952.881408] infiniband mlx5_0: create_qp:3192:(pid 4323): Create QP
> type
> > > 2 failed
> > >
> > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 969.559023] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 969.559027] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 969.559031] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 969.559034] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 969.559036] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000
> > > [ 969.559037] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 969.559039] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 969.559041] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 969.559042] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 969.559044] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 969.559045] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 969.559047] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 969.559049] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [ 969.559051] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe840
> > > [ 969.559052] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 969.559054] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 969.559055] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 969.559057] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 969.559058] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [ 969.559060] cmd[0]: 110: 00000001 d3ae6000 00000002 f1ef7000
> > > [ 969.559062] cmd[0]: 120: 00000003 ae09f000 00000001 28268000
> > > [ 969.559063] cmd[0]: 130: 00000003 ae007000 00000001 b4cf4000
> > > [ 969.559065] cmd[0]: 140: 00000006 180fc000 00000002 2a9e6000
> > > [ 969.559067] cmd[0]: 150: 00000002 f1ed0000 00000001 e54e4000
> > >
> > > [ 969.559070] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 969.559508] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 969.559513] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 969.559517] cmd[0]: 000: 05000000 0065b500 00000000 00000000
> > >
> > > [ 969.559520] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > > [ 969.559553] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 4323):
> > > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5),
> syndrome
> > > (0x65b500)
> > > [ 969.559562] infiniband mlx5_0: create_qp:3192:(pid 4323): Create QP
> type
> > > 2 failed
> > >
> > >
> > > <Test 2 log>
> > > [rdma resource]0: mlx5_0: pd 3 cq 27 qp 261934 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 3543.619746] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 3543.619753] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 3543.619757] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 3543.619759] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 3543.619762] cmd[0]: 020: 00384800 00000015 0003ff5e 00000000
> > > [ 3543.619763] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 3543.619765] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 3543.619796] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 3543.619798] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 3543.619798] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 3543.619799] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 3543.619800] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 3543.619800] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [ 3543.619801] cmd[0]: 0b0: 00000011 0000010d 00000005 22861800
> > > [ 3543.619801] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 3543.619802] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 3543.619803] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 3543.619803] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 3543.619804] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [ 3543.619804] cmd[0]: 110: 00000004 32f72000 00000005 22a11000
> > > [ 3543.619805] cmd[0]: 120: 00000005 229f6000 00000005 22a38000
> > > [ 3543.619806] cmd[0]: 130: 00000001 7244a000 00000002 466aa000
> > > [ 3543.619806] cmd[0]: 140: 00000005 22a19000 00000002 8c046000
> > > [ 3543.619807] cmd[0]: 150: 00000005 8ac7b000 00000002 2c14a000
> > >
> > > [ 3543.619809] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 3543.620284] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 3543.620291] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 3543.620295] cmd[0]: 000: 00000000 00000000 00040122 00000000
> > >
> > > [ 3543.620299] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > >
> > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 3560.087852] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 3560.087857] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 3560.087860] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 3560.087863] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 3560.087865] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000
> > > [ 3560.087867] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 3560.087869] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 3560.087886] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 3560.087887] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 3560.087887] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 3560.087888] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 3560.087888] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 3560.087889] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [ 3560.087890] cmd[0]: 0b0: 00000011 0000010d 00000005 22861840
> > > [ 3560.087890] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 3560.087891] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 3560.087891] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 3560.087892] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 3560.087892] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [ 3560.087893] cmd[0]: 110: 00000005 229e7000 00000004 65ddb000
> > > [ 3560.087894] cmd[0]: 120: 00000005 22a01000 00000005 22a0f000
> > > [ 3560.087894] cmd[0]: 130: 00000005 8ac73000 00000003 a6d42000
> > > [ 3560.087895] cmd[0]: 140: 00000004 05f58000 00000005 22a10000
> > > [ 3560.087896] cmd[0]: 150: 00000005 22a4c000 00000001 fc0da000
> > >
> > > [ 3560.087897] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 3560.088787] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 3560.088793] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 3560.088797] cmd[0]: 000: 05000000 0065b500 00000000 00000000
> > >
> > > [ 3560.088801] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > > [ 3560.088827] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 9217):
> > > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5),
> syndrome
> > > (0x65b500)
> > > [ 3560.088837] infiniband mlx5_0: create_qp:3192:(pid 9217): Create QP
> type
> > > 2 failed
> > >
> > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0
> > >
> > > [ 3577.184180] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915):
> > > cmd[0]: start dump
> > > [ 3577.184185] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [ 3577.184189] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [ 3577.184191] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [ 3577.184194] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000
> > > [ 3577.184195] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [ 3577.184197] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [ 3577.184199] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [ 3577.184200] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [ 3577.184202] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [ 3577.184204] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [ 3577.184205] cmd[0]: 090: 00000000 0000010d 00000000 00000000
> > > [ 3577.184207] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [ 3577.184209] cmd[0]: 0b0: 00000011 0000010d 00000005 22861840
> > > [ 3577.184210] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [ 3577.184212] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [ 3577.184214] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [ 3577.184215] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [ 3577.184217] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [ 3577.184219] cmd[0]: 110: 00000005 229e7000 00000004 65ddb000
> > > [ 3577.184221] cmd[0]: 120: 00000005 22a01000 00000005 22a0f000
> > > [ 3577.184222] cmd[0]: 130: 00000005 8ac73000 00000003 a6d42000
> > > [ 3577.184224] cmd[0]: 140: 00000004 05f58000 00000005 22a10000
> > > [ 3577.184226] cmd[0]: 150: 00000005 22a4c000 00000000 00000000
> > >
> > > [ 3577.184229] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915):
> > > cmd[0]: end dump
> > > [ 3577.184619] mlx5_core 0000:02:00.0: dump_command:844:(pid 0):
> cmd[0]:
> > > start dump
> > > [ 3577.184627] mlx5_core 0000:02:00.0: dump_command:851:(pid 0):
> cmd[0]:
> > > dump command data CREATE_QP(0x500) OUTPUT
> > > [ 3577.184630] cmd[0]: 000: 05000000 0065b500 00000000 00000000
> > >
> > > [ 3577.184635] mlx5_core 0000:02:00.0: dump_command:887:(pid 0):
> cmd[0]:
> > > end dump
> > > [ 3577.184665] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 9217):
> > > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5),
> syndrome
> > > (0x65b500)
> > > [ 3577.184675] infiniband mlx5_0: create_qp:3192:(pid 9217): Create QP
> type
> > > 2 failed
> > >
> > >
> > >
> > > <Test 3 log>
> > > [rdma resource]0: mlx5_0: pd 5 cq 5 qp 130861 cm_id 0 mr 0 ctx 1 srq 2
> > >
> > > [354817.776532] mlx5_core 0000:02:00.0: dump_command:852:(pid
> 103730):
> > > cmd[0]: start dump
> > > [354817.776560] mlx5_core 0000:02:00.0: dump_command:859:(pid
> 103730):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [354817.776583] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [354817.776597] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [354817.776611] cmd[0]: 020: 00384800 00000017 0001ff49 03000000
> > > [354817.776623] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [354817.776955] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [354817.777188] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [354817.777446] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [354817.777701] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [354817.777953] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [354817.778150] cmd[0]: 090: 00000000 00000115 00000000 00000000
> > > [354817.778371] cmd[0]: 0a0: 00000000 00000000 00000800 00000000
> > > [354817.778590] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35ac0
> > > [354817.778824] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [354817.779006] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [354817.779179] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [354817.779377] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [354817.779571] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [354817.779766] cmd[0]: 110: 0000000a 67fb0000 0000000a 67e80000
> > >
> > > [354817.780058] mlx5_core 0000:02:00.0: dump_command:895:(pid
> 103730):
> > > cmd[0]: end dump
> > > [354817.780623] mlx5_core 0000:02:00.0: dump_command:852:(pid 0):
> > > cmd[0]: start dump
> > > [354817.781147] mlx5_core 0000:02:00.0: dump_command:859:(pid 0):
> > > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT
> > > [354817.781478] cmd[0]: 000: 00000000 00000000 00023be6 00000000
> > >
> > > [354817.781893] mlx5_core 0000:02:00.0: dump_command:895:(pid 0):
> > > cmd[0]: end dump
> > >
> > > [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130862 cm_id 0 mr 0 ctx 1 srq 2
> > >
> > > [354834.277537] mlx5_core 0000:02:00.0: dump_command:852:(pid
> 103730):
> > > cmd[0]: start dump
> > > [354834.277755] mlx5_core 0000:02:00.0: dump_command:859:(pid
> 103730):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [354834.277930] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [354834.278108] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [354834.278288] cmd[0]: 020: 00384800 00000017 0001ff4a 03000000
> > > [354834.278499] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [354834.278710] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [354834.278930] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [354834.279102] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [354834.279274] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [354834.279454] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [354834.279685] cmd[0]: 090: 00000000 00000115 00000000 00000000
> > > [354834.279886] cmd[0]: 0a0: 00000000 00000000 00000000 00000000
> > > [354834.280056] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b00
> > > [354834.280226] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [354834.280419] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [354834.280612] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [354834.280819] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [354834.280974] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [354834.281124] cmd[0]: 110: 0000000a 67f98000 0000000a 5f658000
> > >
> > > [354834.281421] mlx5_core 0000:02:00.0: dump_command:895:(pid
> 103730):
> > > cmd[0]: end dump
> > > [354834.282011] mlx5_core 0000:02:00.0: dump_command:852:(pid 0):
> > > cmd[0]: start dump
> > > [354834.282327] mlx5_core 0000:02:00.0: dump_command:859:(pid 0):
> > > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT
> > > [354834.282674] cmd[0]: 000: 00000000 00000000 00023be7 00000000
> > >
> > > [354834.283275] mlx5_core 0000:02:00.0: dump_command:895:(pid 0):
> > > cmd[0]: end dump
> > >
> > > [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130863 cm_id 0 mr 0 ctx 1 srq 2
> > >
> > > [354857.088810] mlx5_core 0000:02:00.0: dump_command:852:(pid
> 103730):
> > > cmd[0]: start dump
> > > [354857.089037] mlx5_core 0000:02:00.0: dump_command:859:(pid
> 103730):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [354857.089216] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [354857.089392] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [354857.089605] cmd[0]: 020: 00384800 00000018 0001ff4b 03000000
> > > [354857.089817] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [354857.090043] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [354857.090221] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [354857.090395] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [354857.090576] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [354857.090809] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [354857.091029] cmd[0]: 090: 00000000 00000115 00000000 00000000
> > > [354857.091206] cmd[0]: 0a0: 00000000 00000000 00000600 00000000
> > > [354857.091382] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b40
> > > [354857.091584] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [354857.091782] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [354857.091993] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [354857.092155] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [354857.092310] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [354857.092469] cmd[0]: 110: 0000000a 5f658000 0000000a 5f788000
> > >
> > > [354857.092829] mlx5_core 0000:02:00.0: dump_command:895:(pid
> 103730):
> > > cmd[0]: end dump
> > > [354857.093900] mlx5_core 0000:02:00.0: dump_command:852:(pid 0):
> > > cmd[0]: start dump
> > > [354857.094313] mlx5_core 0000:02:00.0: dump_command:859:(pid 0):
> > > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT
> > > [354857.094515] cmd[0]: 000: 05000000 0065b500 00000000 00000000
> > >
> > > [354857.094871] mlx5_core 0000:02:00.0: dump_command:895:(pid 0):
> > > cmd[0]: end dump
> > > [354857.095093] mlx5_core 0000:02:00.0: cmd_status_print:808:(pid
> 110379):
> > > CREATE_QP(0x500) op_mod(0x0) uid(2) failed, status bad resource(0x5),
> > > syndrome (0x65b500), err(-22)
> > > [354857.095923] infiniband mlx5_0: create_qp:3035:(pid 110379): Create
> QP
> > > type 2 failed
> > >
> > > [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130863 cm_id 0 mr 0 ctx 1 srq 2
> > >
> > > [354893.848503] mlx5_core 0000:02:00.0: dump_command:852:(pid
> 103730):
> > > cmd[0]: start dump
> > > [354893.848747] mlx5_core 0000:02:00.0: dump_command:859:(pid
> 103730):
> > > cmd[0]: dump command data CREATE_QP(0x500) INPUT
> > > [354893.848949] cmd[0]: 000: 05000002 00000000 00000000 00000000
> > > [354893.849151] cmd[0]: 010: 00000000 00000000 00001800 00000018
> > > [354893.849351] cmd[0]: 020: 00384800 00000018 0001ff4b 03000000
> > > [354893.849562] cmd[0]: 030: 00000000 00000000 00000000 00000000
> > > [354893.849815] cmd[0]: 040: 00000000 00000000 00000000 00000000
> > > [354893.850059] cmd[0]: 050: 00000000 00000000 00000000 00000000
> > > [354893.850258] cmd[0]: 060: 00000000 00000000 00000000 00000000
> > > [354893.850459] cmd[0]: 070: 00000000 00000000 00000000 00000000
> > > [354893.850668] cmd[0]: 080: 00000000 00000000 00000000 00000000
> > > [354893.850939] cmd[0]: 090: 00000000 00000115 00000000 00000000
> > > [354893.851151] cmd[0]: 0a0: 00000000 00000000 00000600 00000000
> > > [354893.851340] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b40
> > > [354893.851526] cmd[0]: 0c0: 00000000 00000101 00000000 00000000
> > > [354893.851740] cmd[0]: 0d0: 00000000 00000000 00000000 00000001
> > > [354893.851948] cmd[0]: 0e0: 00000000 00000000 00000000 00000000
> > > [354893.852165] cmd[0]: 0f0: 00000000 00000000 00000000 00000000
> > > [354893.852339] cmd[0]: 100: 00000000 00000000 00000000 00000000
> > > [354893.852505] cmd[0]: 110: 0000000a 5f658000 0000000a 5f788000
> > >
> > > [354893.852893] mlx5_core 0000:02:00.0: dump_command:895:(pid
> 103730):
> > > cmd[0]: end dump
> > > [354893.853565] mlx5_core 0000:02:00.0: dump_command:852:(pid 0):
> > > cmd[0]: start dump
> > > [354893.854324] mlx5_core 0000:02:00.0: dump_command:859:(pid 0):
> > > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT
> > > [354893.854683] cmd[0]: 000: 05000000 0065b500 00000000 00000000
> > >
> > > [354893.855182] mlx5_core 0000:02:00.0: dump_command:895:(pid 0):
> > > cmd[0]: end dump
> > > [354893.855474] mlx5_core 0000:02:00.0: cmd_status_print:808:(pid
> 110379):
> > > CREATE_QP(0x500) op_mod(0x0) uid(2) failed, status bad resource(0x5),
> > > syndrome (0x65b500), err(-22)
> > > [354893.856328] infiniband mlx5_0: create_qp:3035:(pid 110379): Create
> QP
> > > type 2 failed
> >
