On Thu, 7 Nov 2019 20:52:29 +0000, Parav Pandit wrote: > > On Thu, 7 Nov 2019 10:04:48 -0600, Parav Pandit wrote: > > > Mellanox sub function capability allows users to create several > > > hundreds of networking and/or rdma devices without depending on PCI SR- > > IOV support. > > > > You call the new port type "sub function" but the devlink port flavour is mdev. > > > Sub function is the internal driver structure. The abstract entity at user and stack level is mdev. > Hence the port flavour is mdev. FWIW I agree mdev as flavour seems like the right choice. > > As I'm sure you remember you nacked my patches exposing NFP's PCI sub > > functions which are just regions of the BAR without any mdev capability. Am I > > in the clear to repost those now? Jiri? > > > For sure I didn't nack it. :-) Well, maybe the word "nack" wasn't exactly used :) > What I remember discussing offline/mailing list is > (a) exposing mdev/sub fuctions as devlink sub ports is not so good abstraction > (b) user creating/deleting eswitch sub ports would be hard to fit in the whole usage model Okay, so I can repost the "basic" sub functions? > > > Overview: > > > --------- > > > Mellanox ConnectX sub functions are exposed to user as a mediated > > > device (mdev) [2] as discussed in RFC [3] and further during > > > netdevconf0x13 at [4]. > > > > > > mlx5 mediated device (mdev) enables users to create multiple > > > netdevices and/or RDMA devices from single PCI function. > > > > > > Each mdev maps to a mlx5 sub function. > > > mlx5 sub function is similar to PCI VF. However it doesn't have its > > > own PCI function and MSI-X vectors. > > > > > > mlx5 mdevs share common PCI resources such as PCI BAR region, MSI-X > > > interrupts. > > > > > > Each mdev has its own window in the PCI BAR region, which is > > > accessible only to that mdev and applications using it. > > > > > > Each mlx5 sub function has its own resource namespace for RDMA resources. > > > > > > mdevs are supported when eswitch mode of the devlink instance is in > > > switchdev mode described in devlink documentation [5]. > > > > So presumably the mdevs don't spawn their own devlink instance today, but > > once mapped via VIRTIO to a VM they will create one? > > > mdev doesn't spawn the devlink instance today when mdev is created by user, like PCI. > When PCI bus driver enumerates and creates PCI device, there isn't a devlink instance for it. > > But, mdev's devlink instance is created when mlx5_core driver binds to the mdev device. > (again similar to PCI, when mlx5_core driver binds to PCI, its devlink instance is created ). > > I should have put the example in patch-15 which creates/deletes devlink instance of mdev. > I will revise the commit log of patch-15 to include that. > Good point. Thanks. > > It could be useful to specify. > > > Yes, its certainly useful. I missed to put the example in commit log of patch-15. > > > > Network side: > > > - By default the netdevice and the rdma device of mlx5 mdev cannot > > > send or receive any packets over the network or to any other mlx5 mdev. > > > > Does this mean the frames don't fall back to the repr by default? > Probably I wasn't clear. > What I wanted to say is, that frames transmitted by mdev's netdevice and rdma devices don't go to network. > These frames goes to representor device. > User must configure representor to send/receive/steer traffic to mdev. 👍
