A customer is getting an error creating QPs during scale up testing on Mellanox adapter. I am able to reproduce the issue locally using a test program calling into ibv_create_qp in a loop. The problem does not appear to be related to rdma-core as with different mlx5* drivers, I can either create 260,000+ QPs or 130,000+ QPs. Logs with command buffers and error messages are at the end of this email. All tests are performed using the same server, 64G memory, RHEL 8.5, same ulimit setting and selinux disabled. ConnectX-4 has firmware-version: 12.28.2006 (MT_2180110032), in infiniband mode and max_qp: 262144. Test 1 SW config - RHEL 8.5, MLNX_OFED_LINUX-5.5-1.0.3.2:, max qp created - 261935 Test 2 SW config - same as test 1 with MOFED except rdma-core library is upstream c590cbd51632, max qp created - 261935 (same as test 1) Test 3 SW config - MOFED uninstalled, 5.18.5 kernel installed, rdma-core library is upstream c590cbd51632, max qp created 130863 Can someone help me understand why Test 3 is half of the number? Also, is it possible to hit the max_qp number of 262144? Not to muddle the issue too much, I have observed similar behavior with ConnectX-6 (130763 and 261875 QPs) but don't have logs with those runs. Thanks, Chien I've interlaced "rdma resource" prints in the dmesg log by pausing test program in gdb with breakpoint set to ibv_create_qp. <Test 1 log> [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261933 cm_id 0 mr 0 ctx 1 srq 0 [ 920.711591] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 920.711597] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 920.711601] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 920.711604] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 920.711606] cmd[0]: 020: 00384800 00000015 0003ff5d 04000000 [ 920.711608] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 920.711609] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 920.711611] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 920.711613] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 920.711614] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 920.711616] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 920.711617] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 920.711619] cmd[0]: 0a0: 00000000 00000000 00000300 00000000 [ 920.711621] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe7c0 [ 920.711622] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 920.711624] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 920.711625] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 920.711627] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 920.711629] cmd[0]: 100: 00000001 00000000 00000000 00000000 [ 920.711630] cmd[0]: 110: 00000006 180c0000 00000005 44200000 [ 920.711634] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 920.712125] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 920.712129] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 920.712131] cmd[0]: 000: 00000000 00000000 00040121 00000000 [ 920.712133] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261934 cm_id 0 mr 0 ctx 1 srq 0 [ 938.006403] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 938.006408] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 938.006411] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 938.006414] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 938.006416] cmd[0]: 020: 00384800 00000015 0003ff5e 00000000 [ 938.006418] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 938.006419] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 938.006421] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 938.006422] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 938.006424] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 938.006425] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 938.006427] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 938.006429] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [ 938.006431] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe800 [ 938.006433] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 938.006434] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 938.006436] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 938.006438] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 938.006439] cmd[0]: 100: 00000000 00000000 00000000 00000000 [ 938.006441] cmd[0]: 110: 00000003 5ed0e000 00000003 5ed0f000 [ 938.006443] cmd[0]: 120: 00000006 1820a000 00000002 f1f87000 [ 938.006444] cmd[0]: 130: 00000002 f1fcf000 00000006 1805f000 [ 938.006446] cmd[0]: 140: 00000003 8efe1000 00000006 1805d000 [ 938.006448] cmd[0]: 150: 00000004 956b5000 00000000 00000000 [ 938.006451] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 938.006859] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 938.006864] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 938.006866] cmd[0]: 000: 00000000 00000000 00040122 00000000 [ 938.006869] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 [ 952.880460] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 952.880465] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 952.880468] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 952.880471] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 952.880473] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 [ 952.880475] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 952.880476] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 952.880478] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 952.880480] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 952.880481] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 952.880483] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 952.880485] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 952.880486] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [ 952.880488] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe840 [ 952.880490] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 952.880491] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 952.880493] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 952.880494] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 952.880496] cmd[0]: 100: 00000000 00000000 00000000 00000000 [ 952.880498] cmd[0]: 110: 00000001 d3ae6000 00000002 f1ef7000 [ 952.880499] cmd[0]: 120: 00000003 ae09f000 00000001 28268000 [ 952.880501] cmd[0]: 130: 00000003 ae007000 00000001 b4cf4000 [ 952.880503] cmd[0]: 140: 00000006 180fc000 00000002 2a9e6000 [ 952.880504] cmd[0]: 150: 00000002 f1ed0000 00000002 88295000 [ 952.880508] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 952.881353] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 952.881358] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 952.881360] cmd[0]: 000: 05000000 0065b500 00000000 00000000 [ 952.881363] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [ 952.881401] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 4323): CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome (0x65b500) [ 952.881408] infiniband mlx5_0: create_qp:3192:(pid 4323): Create QP type 2 failed [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 [ 969.559023] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 969.559027] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 969.559031] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 969.559034] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 969.559036] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 [ 969.559037] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 969.559039] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 969.559041] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 969.559042] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 969.559044] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 969.559045] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 969.559047] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 969.559049] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [ 969.559051] cmd[0]: 0b0: 00000011 0000010d 00000005 5b6fe840 [ 969.559052] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 969.559054] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 969.559055] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 969.559057] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 969.559058] cmd[0]: 100: 00000000 00000000 00000000 00000000 [ 969.559060] cmd[0]: 110: 00000001 d3ae6000 00000002 f1ef7000 [ 969.559062] cmd[0]: 120: 00000003 ae09f000 00000001 28268000 [ 969.559063] cmd[0]: 130: 00000003 ae007000 00000001 b4cf4000 [ 969.559065] cmd[0]: 140: 00000006 180fc000 00000002 2a9e6000 [ 969.559067] cmd[0]: 150: 00000002 f1ed0000 00000001 e54e4000 [ 969.559070] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 969.559508] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 969.559513] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 969.559517] cmd[0]: 000: 05000000 0065b500 00000000 00000000 [ 969.559520] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [ 969.559553] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 4323): CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome (0x65b500) [ 969.559562] infiniband mlx5_0: create_qp:3192:(pid 4323): Create QP type 2 failed <Test 2 log> [rdma resource]0: mlx5_0: pd 3 cq 27 qp 261934 cm_id 0 mr 0 ctx 1 srq 0 [ 3543.619746] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 3543.619753] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 3543.619757] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 3543.619759] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 3543.619762] cmd[0]: 020: 00384800 00000015 0003ff5e 00000000 [ 3543.619763] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 3543.619765] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 3543.619796] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 3543.619798] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 3543.619798] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 3543.619799] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 3543.619800] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 3543.619800] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [ 3543.619801] cmd[0]: 0b0: 00000011 0000010d 00000005 22861800 [ 3543.619801] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 3543.619802] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 3543.619803] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 3543.619803] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 3543.619804] cmd[0]: 100: 00000000 00000000 00000000 00000000 [ 3543.619804] cmd[0]: 110: 00000004 32f72000 00000005 22a11000 [ 3543.619805] cmd[0]: 120: 00000005 229f6000 00000005 22a38000 [ 3543.619806] cmd[0]: 130: 00000001 7244a000 00000002 466aa000 [ 3543.619806] cmd[0]: 140: 00000005 22a19000 00000002 8c046000 [ 3543.619807] cmd[0]: 150: 00000005 8ac7b000 00000002 2c14a000 [ 3543.619809] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 3543.620284] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 3543.620291] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 3543.620295] cmd[0]: 000: 00000000 00000000 00040122 00000000 [ 3543.620299] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 [ 3560.087852] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 3560.087857] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 3560.087860] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 3560.087863] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 3560.087865] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 [ 3560.087867] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 3560.087869] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 3560.087886] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 3560.087887] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 3560.087887] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 3560.087888] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 3560.087888] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 3560.087889] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [ 3560.087890] cmd[0]: 0b0: 00000011 0000010d 00000005 22861840 [ 3560.087890] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 3560.087891] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 3560.087891] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 3560.087892] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 3560.087892] cmd[0]: 100: 00000000 00000000 00000000 00000000 [ 3560.087893] cmd[0]: 110: 00000005 229e7000 00000004 65ddb000 [ 3560.087894] cmd[0]: 120: 00000005 22a01000 00000005 22a0f000 [ 3560.087894] cmd[0]: 130: 00000005 8ac73000 00000003 a6d42000 [ 3560.087895] cmd[0]: 140: 00000004 05f58000 00000005 22a10000 [ 3560.087896] cmd[0]: 150: 00000005 22a4c000 00000001 fc0da000 [ 3560.087897] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 3560.088787] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 3560.088793] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 3560.088797] cmd[0]: 000: 05000000 0065b500 00000000 00000000 [ 3560.088801] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [ 3560.088827] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 9217): CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome (0x65b500) [ 3560.088837] infiniband mlx5_0: create_qp:3192:(pid 9217): Create QP type 2 failed [rdma resource] 0: mlx5_0: pd 3 cq 27 qp 261935 cm_id 0 mr 0 ctx 1 srq 0 [ 3577.184180] mlx5_core 0000:02:00.0: dump_command:844:(pid 1915): cmd[0]: start dump [ 3577.184185] mlx5_core 0000:02:00.0: dump_command:851:(pid 1915): cmd[0]: dump command data CREATE_QP(0x500) INPUT [ 3577.184189] cmd[0]: 000: 05000002 00000000 00000000 00000000 [ 3577.184191] cmd[0]: 010: 00000000 00000000 00001800 00000018 [ 3577.184194] cmd[0]: 020: 00384800 00000016 0003ff5f 00000000 [ 3577.184195] cmd[0]: 030: 00000000 00000000 00000000 00000000 [ 3577.184197] cmd[0]: 040: 00000000 00000000 00000000 00000000 [ 3577.184199] cmd[0]: 050: 00000000 00000000 00000000 00000000 [ 3577.184200] cmd[0]: 060: 00000000 00000000 00000000 00000000 [ 3577.184202] cmd[0]: 070: 00000000 00000000 00000000 00000000 [ 3577.184204] cmd[0]: 080: 00000000 00000000 00000000 00000000 [ 3577.184205] cmd[0]: 090: 00000000 0000010d 00000000 00000000 [ 3577.184207] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [ 3577.184209] cmd[0]: 0b0: 00000011 0000010d 00000005 22861840 [ 3577.184210] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [ 3577.184212] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [ 3577.184214] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [ 3577.184215] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [ 3577.184217] cmd[0]: 100: 00000000 00000000 00000000 00000000 [ 3577.184219] cmd[0]: 110: 00000005 229e7000 00000004 65ddb000 [ 3577.184221] cmd[0]: 120: 00000005 22a01000 00000005 22a0f000 [ 3577.184222] cmd[0]: 130: 00000005 8ac73000 00000003 a6d42000 [ 3577.184224] cmd[0]: 140: 00000004 05f58000 00000005 22a10000 [ 3577.184226] cmd[0]: 150: 00000005 22a4c000 00000000 00000000 [ 3577.184229] mlx5_core 0000:02:00.0: dump_command:887:(pid 1915): cmd[0]: end dump [ 3577.184619] mlx5_core 0000:02:00.0: dump_command:844:(pid 0): cmd[0]: start dump [ 3577.184627] mlx5_core 0000:02:00.0: dump_command:851:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [ 3577.184630] cmd[0]: 000: 05000000 0065b500 00000000 00000000 [ 3577.184635] mlx5_core 0000:02:00.0: dump_command:887:(pid 0): cmd[0]: end dump [ 3577.184665] mlx5_core 0000:02:00.0: mlx5_cmd_check:827:(pid 9217): CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome (0x65b500) [ 3577.184675] infiniband mlx5_0: create_qp:3192:(pid 9217): Create QP type 2 failed <Test 3 log> [rdma resource]0: mlx5_0: pd 5 cq 5 qp 130861 cm_id 0 mr 0 ctx 1 srq 2 [354817.776532] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): cmd[0]: start dump [354817.776560] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): cmd[0]: dump command data CREATE_QP(0x500) INPUT [354817.776583] cmd[0]: 000: 05000002 00000000 00000000 00000000 [354817.776597] cmd[0]: 010: 00000000 00000000 00001800 00000018 [354817.776611] cmd[0]: 020: 00384800 00000017 0001ff49 03000000 [354817.776623] cmd[0]: 030: 00000000 00000000 00000000 00000000 [354817.776955] cmd[0]: 040: 00000000 00000000 00000000 00000000 [354817.777188] cmd[0]: 050: 00000000 00000000 00000000 00000000 [354817.777446] cmd[0]: 060: 00000000 00000000 00000000 00000000 [354817.777701] cmd[0]: 070: 00000000 00000000 00000000 00000000 [354817.777953] cmd[0]: 080: 00000000 00000000 00000000 00000000 [354817.778150] cmd[0]: 090: 00000000 00000115 00000000 00000000 [354817.778371] cmd[0]: 0a0: 00000000 00000000 00000800 00000000 [354817.778590] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35ac0 [354817.778824] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [354817.779006] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [354817.779179] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [354817.779377] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [354817.779571] cmd[0]: 100: 00000000 00000000 00000000 00000000 [354817.779766] cmd[0]: 110: 0000000a 67fb0000 0000000a 67e80000 [354817.780058] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): cmd[0]: end dump [354817.780623] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): cmd[0]: start dump [354817.781147] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [354817.781478] cmd[0]: 000: 00000000 00000000 00023be6 00000000 [354817.781893] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): cmd[0]: end dump [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130862 cm_id 0 mr 0 ctx 1 srq 2 [354834.277537] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): cmd[0]: start dump [354834.277755] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): cmd[0]: dump command data CREATE_QP(0x500) INPUT [354834.277930] cmd[0]: 000: 05000002 00000000 00000000 00000000 [354834.278108] cmd[0]: 010: 00000000 00000000 00001800 00000018 [354834.278288] cmd[0]: 020: 00384800 00000017 0001ff4a 03000000 [354834.278499] cmd[0]: 030: 00000000 00000000 00000000 00000000 [354834.278710] cmd[0]: 040: 00000000 00000000 00000000 00000000 [354834.278930] cmd[0]: 050: 00000000 00000000 00000000 00000000 [354834.279102] cmd[0]: 060: 00000000 00000000 00000000 00000000 [354834.279274] cmd[0]: 070: 00000000 00000000 00000000 00000000 [354834.279454] cmd[0]: 080: 00000000 00000000 00000000 00000000 [354834.279685] cmd[0]: 090: 00000000 00000115 00000000 00000000 [354834.279886] cmd[0]: 0a0: 00000000 00000000 00000000 00000000 [354834.280056] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b00 [354834.280226] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [354834.280419] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [354834.280612] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [354834.280819] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [354834.280974] cmd[0]: 100: 00000000 00000000 00000000 00000000 [354834.281124] cmd[0]: 110: 0000000a 67f98000 0000000a 5f658000 [354834.281421] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): cmd[0]: end dump [354834.282011] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): cmd[0]: start dump [354834.282327] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [354834.282674] cmd[0]: 000: 00000000 00000000 00023be7 00000000 [354834.283275] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): cmd[0]: end dump [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130863 cm_id 0 mr 0 ctx 1 srq 2 [354857.088810] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): cmd[0]: start dump [354857.089037] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): cmd[0]: dump command data CREATE_QP(0x500) INPUT [354857.089216] cmd[0]: 000: 05000002 00000000 00000000 00000000 [354857.089392] cmd[0]: 010: 00000000 00000000 00001800 00000018 [354857.089605] cmd[0]: 020: 00384800 00000018 0001ff4b 03000000 [354857.089817] cmd[0]: 030: 00000000 00000000 00000000 00000000 [354857.090043] cmd[0]: 040: 00000000 00000000 00000000 00000000 [354857.090221] cmd[0]: 050: 00000000 00000000 00000000 00000000 [354857.090395] cmd[0]: 060: 00000000 00000000 00000000 00000000 [354857.090576] cmd[0]: 070: 00000000 00000000 00000000 00000000 [354857.090809] cmd[0]: 080: 00000000 00000000 00000000 00000000 [354857.091029] cmd[0]: 090: 00000000 00000115 00000000 00000000 [354857.091206] cmd[0]: 0a0: 00000000 00000000 00000600 00000000 [354857.091382] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b40 [354857.091584] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [354857.091782] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [354857.091993] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [354857.092155] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [354857.092310] cmd[0]: 100: 00000000 00000000 00000000 00000000 [354857.092469] cmd[0]: 110: 0000000a 5f658000 0000000a 5f788000 [354857.092829] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): cmd[0]: end dump [354857.093900] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): cmd[0]: start dump [354857.094313] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [354857.094515] cmd[0]: 000: 05000000 0065b500 00000000 00000000 [354857.094871] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): cmd[0]: end dump [354857.095093] mlx5_core 0000:02:00.0: cmd_status_print:808:(pid 110379): CREATE_QP(0x500) op_mod(0x0) uid(2) failed, status bad resource(0x5), syndrome (0x65b500), err(-22) [354857.095923] infiniband mlx5_0: create_qp:3035:(pid 110379): Create QP type 2 failed [rdma resource] 0: mlx5_0: pd 5 cq 5 qp 130863 cm_id 0 mr 0 ctx 1 srq 2 [354893.848503] mlx5_core 0000:02:00.0: dump_command:852:(pid 103730): cmd[0]: start dump [354893.848747] mlx5_core 0000:02:00.0: dump_command:859:(pid 103730): cmd[0]: dump command data CREATE_QP(0x500) INPUT [354893.848949] cmd[0]: 000: 05000002 00000000 00000000 00000000 [354893.849151] cmd[0]: 010: 00000000 00000000 00001800 00000018 [354893.849351] cmd[0]: 020: 00384800 00000018 0001ff4b 03000000 [354893.849562] cmd[0]: 030: 00000000 00000000 00000000 00000000 [354893.849815] cmd[0]: 040: 00000000 00000000 00000000 00000000 [354893.850059] cmd[0]: 050: 00000000 00000000 00000000 00000000 [354893.850258] cmd[0]: 060: 00000000 00000000 00000000 00000000 [354893.850459] cmd[0]: 070: 00000000 00000000 00000000 00000000 [354893.850668] cmd[0]: 080: 00000000 00000000 00000000 00000000 [354893.850939] cmd[0]: 090: 00000000 00000115 00000000 00000000 [354893.851151] cmd[0]: 0a0: 00000000 00000000 00000600 00000000 [354893.851340] cmd[0]: 0b0: 00000011 00000115 0000000a 67c35b40 [354893.851526] cmd[0]: 0c0: 00000000 00000101 00000000 00000000 [354893.851740] cmd[0]: 0d0: 00000000 00000000 00000000 00000001 [354893.851948] cmd[0]: 0e0: 00000000 00000000 00000000 00000000 [354893.852165] cmd[0]: 0f0: 00000000 00000000 00000000 00000000 [354893.852339] cmd[0]: 100: 00000000 00000000 00000000 00000000 [354893.852505] cmd[0]: 110: 0000000a 5f658000 0000000a 5f788000 [354893.852893] mlx5_core 0000:02:00.0: dump_command:895:(pid 103730): cmd[0]: end dump [354893.853565] mlx5_core 0000:02:00.0: dump_command:852:(pid 0): cmd[0]: start dump [354893.854324] mlx5_core 0000:02:00.0: dump_command:859:(pid 0): cmd[0]: dump command data CREATE_QP(0x500) OUTPUT [354893.854683] cmd[0]: 000: 05000000 0065b500 00000000 00000000 [354893.855182] mlx5_core 0000:02:00.0: dump_command:895:(pid 0): cmd[0]: end dump [354893.855474] mlx5_core 0000:02:00.0: cmd_status_print:808:(pid 110379): CREATE_QP(0x500) op_mod(0x0) uid(2) failed, status bad resource(0x5), syndrome (0x65b500), err(-22) [354893.856328] infiniband mlx5_0: create_qp:3035:(pid 110379): Create QP type 2 failed
