Hello,
I am running latest firmware on a QLogic/Cavium/Marvell QL41112HLRJ,
and inbox Linux drivers. I'm using Ubuntu 21.04, kernel
5.11.0-37-generic.
I am trying to connect to an SMB share on a Windows Server 2019
machine from the Ubuntu machine, but am unable to get an RDMA
connection going. After having recompiled the kernel with
CONFIG_CIFS_SMB_DIRECT=y (default is n), trying to connect with
$ sudo mount -t cifs --verbose -o
rdma,multichannel,vers=3.1.1,username=***,password=***
//10.0.1.45/scratch /mnt/test
i get:
$ dmesg:
[ 1484.306940] CIFS: Attempting to mount \\10.0.1.45\scratch
[ 1484.310637] CIFS: VFS: _smbd_get_connection:1556 warning: device
max_send_sge = 4 too small
[ 1484.310647] CIFS: VFS: _smbd_get_connection:1559 Queue Pair creation may fail
[ 1484.310651] CIFS: VFS: _smbd_get_connection:1562 warning: device
max_recv_sge = 4 too small
[ 1484.310655] CIFS: VFS: _smbd_get_connection:1565 Queue Pair creation may fail
[ 1484.310793] [qedr_check_qp_attrs:1230(qedr0)]create qp: unsupported
send_sge=0x10 requested (max_send_sge=0x4)
[ 1484.310799] CIFS: VFS: _smbd_get_connection:1602 rdma_create_qp failed -22
Is there a way i can increase this max_send_sge under Linux, or is
there some other (e.g. cifs) configuration i should do? SMB Direct
works well Windows Server -> Windows Server.
Thanks and all the best,
Dee
Further info that may be relevant:
$ dmesg | grep qed
[ 1.408266] QLogic FastLinQ 4xxxx Core Module qed 8.37.0.20
[ 1.416589] qede_init: QLogic FastLinQ 4xxxx Ethernet Driver qede 8.37.0.20
[ 1.768448] qede 0000:19:00.0: no suspend buffer for PTM
[ 2.061021] qede 19:00.00: Storm FW 8.42.2.0, Management FW
8.55.43.0 [MBI 8.55.24] [eth0]
[ 2.148912] qede 19:00.01: Storm FW 8.42.2.0, Management FW
8.55.43.0 [MBI 8.55.24] [eth1]
[ 2.157843] qede 0000:19:00.0 enp25s0f0: renamed from eth0
[ 2.188371] qede 0000:19:00.1 enp25s0f1: renamed from eth1
[ 8.917308] [qede_link_update:2495(enp25s0f0)]Link is up
[ 8.947363] [qede_link_update:2495(enp25s0f1)]Link is up
[ 1481.149961] qedr: discovered and registered 2 RDMA funcs
$ lsmod | grep qed
qedr 110592 0
ib_uverbs 151552 2 rdma_ucm,qedr
ib_core 364544 6 rdma_cm,iw_cm,rdma_ucm,ib_uverbs,qedr,ib_cm
qede 151552 1 qedr
qed 700416 2 qede,qedr
crc8 16384 1 qed
$ mount.cifs -V
mount.cifs version: 6.11
$ ibv_devinfo -d qedr0 -v
hca_id: qedr0
transport: iWARP (1)
fw_ver: 8.42.2.0
node_guid: f6e9:d4ff:fe73:6fec
sys_image_guid: f6e9:d4ff:fe73:6fec
vendor_id: 0x1077
vendor_part_id: 32880
hw_ver: 0x0
phys_port_cnt: 1
max_mr_size: 0x10000000000
page_size_cap: 0xfffff000
max_qp: 7936
max_qp_wr: 32767
device_cap_flags: 0x00209080
CURR_QP_STATE_MOD
RC_RNR_NAK_GEN
MEM_MGT_EXTENSIONS
Unknown flags: 0x8000
max_sge: 4
max_sge_rd: 4
max_cq: 7936
max_cqe: 8388480
max_mr: 131070
max_pd: 65536
max_qp_rd_atom: 32
max_ee_rd_atom: 0
max_res_rd_atom: 0
max_qp_init_rd_atom: 32
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_GLOB (2)
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 0
max_mcast_qp_attach: 0
max_total_mcast_qp_attach: 0
max_ah: 8192
max_fmr: 0
max_srq: 8192
max_srq_wr: 32767
max_srq_sge: 0
max_pkeys: 0
local_ca_ack_delay: 15
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
completion_timestamp_mask not supported
core clock not supported
device_cap_flags_ex: 0x209080
tso_caps:
max_tso: 0
rss_caps:
max_rwq_indirection_tables: 0
max_rwq_indirection_table_size: 0
rx_hash_function: 0x0
rx_hash_fields_mask: 0x0
max_wq_type_rq: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
tag matching not supported
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x80000000
port_cap_flags: 0x04000000
port_cap_flags2: 0x0000
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 0
gid_tbl_len: 1
subnet_timeout: 0
init_type_reply: 0
active_width: 1X (1)
active_speed: 10.0 Gbps (4)
