No one is interested in helping with Create QP issue with Mellanox controller? > -----Original Message----- > From: Tung, Chien Tin <chien.tin.tung@xxxxxxxxx> > Sent: Monday, June 27, 2022 6:05 PM > To: linux-rdma@xxxxxxxxxxxxxxx > Subject: Unable to create max_qp QPs on Mellanox ConnectX-4/6 > > A customer is getting an error creating QPs during scale up testing on > Mellanox adapter. > I am able to reproduce the issue locally using a test program calling into > ibv_create_qp in a loop. > The problem does not appear to be related to rdma-core as with different > mlx5* drivers, I can either create 260,000+ QPs or 130,000+ QPs. > > Logs with command buffers and error messages are at the end of this email. > All tests are performed using the same server, 64G memory, RHEL 8.5, same > ulimit setting and selinux disabled. > ConnectX-4 has firmware-version: 12.28.2006 (MT_2180110032), in infiniband > mode and max_qp: 262144. > > Test 1 SW config - RHEL 8.5, MLNX_OFED_LINUX-5.5-1.0.3.2:, max qp created > - 261935 > Test 2 SW config - same as test 1 with MOFED except rdma-core library is > upstream c590cbd51632, max qp created - 261935 (same as test 1) > Test 3 SW config - MOFED uninstalled, 5.18.5 kernel installed, rdma-core > library is upstream c590cbd51632, max qp created 130863 > > Can someone help me understand why Test 3 is half of the number? Also, is it > possible to hit the max_qp number of 262144? > Not to muddle the issue too much, I have observed similar behavior with > ConnectX-6 (130763 and 261875 QPs) but don't have logs with those runs. > > Thanks, > > Chien > > I've interlaced "rdma resource" prints in the dmesg log by pausing test > program in gdb with breakpoint set to ibv_create_qp. > > <Test 1 log> > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261933 cm_id 0 mr 0 ctx 1 srq 0 > > [ 920.711591] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 920.711597] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 920.711601] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 920.711604] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 920.711606] cmd[0]: 020: 00384800 00000015 0003ff5d 04000000 > [ 920.711608] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 920.711609] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 920.711611] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 920.711613] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 920.711614] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 920.711616] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 920.711617] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 920.711619] cmd[0]: 0a0: 00000000 00000000 00000300 00000000 > [ 920.711621] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe7c0 > [ 920.711622] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 920.711624] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 920.711625] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 920.711627] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 920.711629] cmd[0]: 100: 00000001 00000000 00000000 00000000 > [ 920.711630] cmd[0]: 110: 00000006 180c0000 00000005 44200000 > > [ 920.711634] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 920.712125] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 920.712129] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 920.712131] cmd[0]: 000: 00000000 00000000 00040121 00000000 > > [ 920.712133] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261934 cm_id 0 mr 0 ctx 1 srq 0 > > [ 938.006403] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 938.006408] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 938.006411] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 938.006414] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 938.006416] cmd[0]: 020: 00384800 00000015 0003ff5e 00000000 > [ 938.006418] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 938.006419] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 938.006421] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 938.006422] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 938.006424] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 938.006425] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 938.006427] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 938.006429] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [ 938.006431] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe800 > [ 938.006433] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 938.006434] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 938.006436] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 938.006438] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 938.006439] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [ 938.006441] cmd[0]: 110: 00000003 5ed0e000 00000003 5ed0f000 > [ 938.006443] cmd[0]: 120: 00000006 1820a000 00000002 f1f87000 > [ 938.006444] cmd[0]: 130: 00000002 f1fcf000 00000006 1805f000 > [ 938.006446] cmd[0]: 140: 00000003 8efe1000 00000006 1805d000 > [ 938.006448] cmd[0]: 150: 00000004 956b5000 00000000 00000000 > > [ 938.006451] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 938.006859] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 938.006864] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 938.006866] cmd[0]: 000: 00000000 00000000 00040122 00000000 > > [ 938.006869] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 > > [ 952.880460] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 952.880465] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 952.880468] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 952.880471] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 952.880473] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 > [ 952.880475] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 952.880476] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 952.880478] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 952.880480] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 952.880481] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 952.880483] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 952.880485] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 952.880486] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [ 952.880488] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe840 > [ 952.880490] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 952.880491] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 952.880493] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 952.880494] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 952.880496] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [ 952.880498] cmd[0]: 110: 00000001 d3ae6000 00000002 f1ef7000 > [ 952.880499] cmd[0]: 120: 00000003 ae09f000 00000001 28268000 > [ 952.880501] cmd[0]: 130: 00000003 ae007000 00000001 b4cf4000 > [ 952.880503] cmd[0]: 140: 00000006 180fc000 00000002 2a9e6000 > [ 952.880504] cmd[0]: 150: 00000002 f1ed0000 00000002 88295000 > > [ 952.880508] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 952.881353] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 952.881358] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 952.881360] cmd[0]: 000: 05000000 0065b500 00000000 00000000 > > [ 952.881363] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > [ 952.881401] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 4323): > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome > (0x65b500) > [ 952.881408] infiniband mlx5_0: create_qp:3192:(pid 4323): Create QP type > 2 failed > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 > > [ 969.559023] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 969.559027] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 969.559031] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 969.559034] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 969.559036] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 > [ 969.559037] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 969.559039] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 969.559041] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 969.559042] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 969.559044] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 969.559045] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 969.559047] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 969.559049] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [ 969.559051] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe840 > [ 969.559052] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 969.559054] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 969.559055] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 969.559057] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 969.559058] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [ 969.559060] cmd[0]: 110: 00000001 d3ae6000 00000002 f1ef7000 > [ 969.559062] cmd[0]: 120: 00000003 ae09f000 00000001 28268000 > [ 969.559063] cmd[0]: 130: 00000003 ae007000 00000001 b4cf4000 > [ 969.559065] cmd[0]: 140: 00000006 180fc000 00000002 2a9e6000 > [ 969.559067] cmd[0]: 150: 00000002 f1ed0000 00000001 e54e4000 > > [ 969.559070] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 969.559508] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 969.559513] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 969.559517] cmd[0]: 000: 05000000 0065b500 00000000 00000000 > > [ 969.559520] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > [ 969.559553] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 4323): > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome > (0x65b500) > [ 969.559562] infiniband mlx5_0: create_qp:3192:(pid 4323): Create QP type > 2 failed > > > <Test 2 log> > [rdma resource]0: mlx5_0: pd 3 cq 27 qp 261934 cm_id 0 mr 0 ctx 1 srq 0 > > [ 3543.619746] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 3543.619753] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 3543.619757] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 3543.619759] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 3543.619762] cmd[0]: 020: 00384800 00000015 0003ff5e 00000000 > [ 3543.619763] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 3543.619765] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 3543.619796] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 3543.619798] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 3543.619798] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 3543.619799] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 3543.619800] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 3543.619800] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [ 3543.619801] cmd[0]: 0b0: 00000011 0000010d 00000005 22861800 > [ 3543.619801] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 3543.619802] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 3543.619803] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 3543.619803] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 3543.619804] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [ 3543.619804] cmd[0]: 110: 00000004 32f72000 00000005 22a11000 > [ 3543.619805] cmd[0]: 120: 00000005 229f6000 00000005 22a38000 > [ 3543.619806] cmd[0]: 130: 00000001 7244a000 00000002 466aa000 > [ 3543.619806] cmd[0]: 140: 00000005 22a19000 00000002 8c046000 > [ 3543.619807] cmd[0]: 150: 00000005 8ac7b000 00000002 2c14a000 > > [ 3543.619809] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 3543.620284] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 3543.620291] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 3543.620295] cmd[0]: 000: 00000000 00000000 00040122 00000000 > > [ 3543.620299] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 > > [ 3560.087852] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 3560.087857] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 3560.087860] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 3560.087863] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 3560.087865] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 > [ 3560.087867] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 3560.087869] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 3560.087886] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 3560.087887] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 3560.087887] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 3560.087888] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 3560.087888] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 3560.087889] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [ 3560.087890] cmd[0]: 0b0: 00000011 0000010d 00000005 22861840 > [ 3560.087890] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 3560.087891] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 3560.087891] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 3560.087892] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 3560.087892] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [ 3560.087893] cmd[0]: 110: 00000005 229e7000 00000004 65ddb000 > [ 3560.087894] cmd[0]: 120: 00000005 22a01000 00000005 22a0f000 > [ 3560.087894] cmd[0]: 130: 00000005 8ac73000 00000003 a6d42000 > [ 3560.087895] cmd[0]: 140: 00000004 05f58000 00000005 22a10000 > [ 3560.087896] cmd[0]: 150: 00000005 22a4c000 00000001 fc0da000 > > [ 3560.087897] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 3560.088787] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 3560.088793] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 3560.088797] cmd[0]: 000: 05000000 0065b500 00000000 00000000 > > [ 3560.088801] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > [ 3560.088827] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 9217): > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome > (0x65b500) > [ 3560.088837] infiniband mlx5_0: create_qp:3192:(pid 9217): Create QP type > 2 failed > > [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 > > [ 3577.184180] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): > cmd[0]: start dump > [ 3577.184185] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [ 3577.184189] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [ 3577.184191] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [ 3577.184194] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 > [ 3577.184195] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [ 3577.184197] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [ 3577.184199] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [ 3577.184200] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [ 3577.184202] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [ 3577.184204] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [ 3577.184205] cmd[0]: 090: 00000000 0000010d 00000000 00000000 > [ 3577.184207] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [ 3577.184209] cmd[0]: 0b0: 00000011 0000010d 00000005 22861840 > [ 3577.184210] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [ 3577.184212] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [ 3577.184214] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [ 3577.184215] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [ 3577.184217] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [ 3577.184219] cmd[0]: 110: 00000005 229e7000 00000004 65ddb000 > [ 3577.184221] cmd[0]: 120: 00000005 22a01000 00000005 22a0f000 > [ 3577.184222] cmd[0]: 130: 00000005 8ac73000 00000003 a6d42000 > [ 3577.184224] cmd[0]: 140: 00000004 05f58000 00000005 22a10000 > [ 3577.184226] cmd[0]: 150: 00000005 22a4c000 00000000 00000000 > > [ 3577.184229] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): > cmd[0]: end dump > [ 3577.184619] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: > start dump > [ 3577.184627] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: > dump command data CREATE_QP(0x500) OUTPUT > [ 3577.184630] cmd[0]: 000: 05000000 0065b500 00000000 00000000 > > [ 3577.184635] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: > end dump > [ 3577.184665] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 9217): > CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome > (0x65b500) > [ 3577.184675] infiniband mlx5_0: create_qp:3192:(pid 9217): Create QP type > 2 failed > > > > <Test 3 log> > [rdma resource]0: mlx5_0: pd 5 cq 5 qp 130861 cm_id 0 mr 0 ctx 1 srq 2 > > [354817.776532] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): > cmd[0]: start dump > [354817.776560] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [354817.776583] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [354817.776597] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [354817.776611] cmd[0]: 020: 00384800 00000017 0001ff49 03000000 > [354817.776623] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [354817.776955] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [354817.777188] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [354817.777446] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [354817.777701] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [354817.777953] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [354817.778150] cmd[0]: 090: 00000000 00000115 00000000 00000000 > [354817.778371] cmd[0]: 0a0: 00000000 00000000 00000800 00000000 > [354817.778590] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35ac0 > [354817.778824] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [354817.779006] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [354817.779179] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [354817.779377] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [354817.779571] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [354817.779766] cmd[0]: 110: 0000000a 67fb0000 0000000a 67e80000 > > [354817.780058] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): > cmd[0]: end dump > [354817.780623] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): > cmd[0]: start dump > [354817.781147] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT > [354817.781478] cmd[0]: 000: 00000000 00000000 00023be6 00000000 > > [354817.781893] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): > cmd[0]: end dump > > [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130862 cm_id 0 mr 0 ctx 1 srq 2 > > [354834.277537] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): > cmd[0]: start dump > [354834.277755] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [354834.277930] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [354834.278108] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [354834.278288] cmd[0]: 020: 00384800 00000017 0001ff4a 03000000 > [354834.278499] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [354834.278710] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [354834.278930] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [354834.279102] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [354834.279274] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [354834.279454] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [354834.279685] cmd[0]: 090: 00000000 00000115 00000000 00000000 > [354834.279886] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 > [354834.280056] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b00 > [354834.280226] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [354834.280419] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [354834.280612] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [354834.280819] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [354834.280974] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [354834.281124] cmd[0]: 110: 0000000a 67f98000 0000000a 5f658000 > > [354834.281421] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): > cmd[0]: end dump > [354834.282011] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): > cmd[0]: start dump > [354834.282327] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT > [354834.282674] cmd[0]: 000: 00000000 00000000 00023be7 00000000 > > [354834.283275] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): > cmd[0]: end dump > > [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130863 cm_id 0 mr 0 ctx 1 srq 2 > > [354857.088810] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): > cmd[0]: start dump > [354857.089037] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [354857.089216] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [354857.089392] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [354857.089605] cmd[0]: 020: 00384800 00000018 0001ff4b 03000000 > [354857.089817] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [354857.090043] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [354857.090221] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [354857.090395] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [354857.090576] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [354857.090809] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [354857.091029] cmd[0]: 090: 00000000 00000115 00000000 00000000 > [354857.091206] cmd[0]: 0a0: 00000000 00000000 00000600 00000000 > [354857.091382] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b40 > [354857.091584] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [354857.091782] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [354857.091993] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [354857.092155] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [354857.092310] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [354857.092469] cmd[0]: 110: 0000000a 5f658000 0000000a 5f788000 > > [354857.092829] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): > cmd[0]: end dump > [354857.093900] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): > cmd[0]: start dump > [354857.094313] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT > [354857.094515] cmd[0]: 000: 05000000 0065b500 00000000 00000000 > > [354857.094871] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): > cmd[0]: end dump > [354857.095093] mlx5_core 0000:02:00.0: cmd_status_print:808:(pid 110379): > CREATE_QP(0x500) op_mod(0x0) uid(2) failed, status bad resource(0x5), > syndrome (0x65b500), err(-22) > [354857.095923] infiniband mlx5_0: create_qp:3035:(pid 110379): Create QP > type 2 failed > > [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130863 cm_id 0 mr 0 ctx 1 srq 2 > > [354893.848503] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): > cmd[0]: start dump > [354893.848747] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): > cmd[0]: dump command data CREATE_QP(0x500) INPUT > [354893.848949] cmd[0]: 000: 05000002 00000000 00000000 00000000 > [354893.849151] cmd[0]: 010: 00000000 00000000 00001800 00000018 > [354893.849351] cmd[0]: 020: 00384800 00000018 0001ff4b 03000000 > [354893.849562] cmd[0]: 030: 00000000 00000000 00000000 00000000 > [354893.849815] cmd[0]: 040: 00000000 00000000 00000000 00000000 > [354893.850059] cmd[0]: 050: 00000000 00000000 00000000 00000000 > [354893.850258] cmd[0]: 060: 00000000 00000000 00000000 00000000 > [354893.850459] cmd[0]: 070: 00000000 00000000 00000000 00000000 > [354893.850668] cmd[0]: 080: 00000000 00000000 00000000 00000000 > [354893.850939] cmd[0]: 090: 00000000 00000115 00000000 00000000 > [354893.851151] cmd[0]: 0a0: 00000000 00000000 00000600 00000000 > [354893.851340] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b40 > [354893.851526] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 > [354893.851740] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 > [354893.851948] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 > [354893.852165] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 > [354893.852339] cmd[0]: 100: 00000000 00000000 00000000 00000000 > [354893.852505] cmd[0]: 110: 0000000a 5f658000 0000000a 5f788000 > > [354893.852893] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): > cmd[0]: end dump > [354893.853565] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): > cmd[0]: start dump > [354893.854324] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): > cmd[0]: dump command data CREATE_QP(0x500) OUTPUT > [354893.854683] cmd[0]: 000: 05000000 0065b500 00000000 00000000 > > [354893.855182] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): > cmd[0]: end dump > [354893.855474] mlx5_core 0000:02:00.0: cmd_status_print:808:(pid 110379): > CREATE_QP(0x500) op_mod(0x0) uid(2) failed, status bad resource(0x5), > syndrome (0x65b500), err(-22) > [354893.856328] infiniband mlx5_0: create_qp:3035:(pid 110379): Create QP > type 2 failed
