On Wed, May 01, 2024 at 04:43:39AM +0000, Ewen Chan wrote: > Leon: > > I see in the link that you provided that the IB management packages are only available for Redhat. > > Would it be safe to assume that there are no Debian equivalent to these packages? Right, at the moment, there is no Debian equivalent to these packages. Thanks > > Your help is greatly appreciated. > > Thank you. > > > > > From: Leon Romanovsky <leon@xxxxxxxxxx> > Sent: April 30, 2024 8:03 AM > To: Ewen Chan <alpha754293@xxxxxxxxxxx> > Cc: linux-rdma@xxxxxxxxxxxxxxx <linux-rdma@xxxxxxxxxxxxxxx> > Subject: Re: SRIOV virtual functions with Linux "inbox" opensm > > On Mon, Apr 22, 2024 at 10:09:15AM +0000, Ewen Chan wrote: > > To Whom It May Concern: > > > > I am using a few Mellanox ConnectX-4 100 Gbps Infiniband NIC that's connected together via a Mellanox MSB7890 externally managed switch. > > > > I have a dual Xeon E5-2697A v4, Proxmox 7.4-17 (Debian 11) server that's running opensm, along with two AMD Ryzen 5950X compute nodes, that also have the ConnectX-4 in them, running Proxmox 7.4-17 as well. > > > > I have enabled SR-IOV on all three systems, and all three systems have 8 virtual functions for said ConnectX-4. > > > > I read in the Nvidia/Mellanox documentation that I would need to add the parameter "virt_enabled 2" to /etc/opensm/opensm.conf so that the OpenSM subnet manager will know that virtual functions are enabled, but it would appear that the opensm that ships with Debian 11/linux-rdma, either ignores that option or doesn't know what to do with it. > > > > I would prefer NOT to install the MLNX_OFED drivers for Debian (11) if I can avoid it. > > > > My two questions are how do I get the linux opensm to: > > > > Recognise that I am using virtual functions (so that it would understand that there are multiple traffic streams coming over the wire, via one physical port)? > > > > Automatically assign the Node GUID and Port GUID so that I don't have to set those manually. > > > > (I've set the Node GUID and Port GUID on the my Ryzen compute node host already, and I can see the Node GUID and Port GUID inside my CentOS 7.7.1908 VM (which I've updated to use the 5.4.247 kernel), but it is still showing "Port 1, State: Down".) > > > > > > Your help is greatly appreciated. > > Linux opensm is not supporting ConnectX4+ SRIOV. > > You can install SM RPM from NVIDIA Web to enable ConnectX4 SRIOV. > https://network.nvidia.com/products/adapter-software/infiniband-management-and-monitoring-tools/ > > Thanks > > > > > Thank you. > >
