> -----Original Message----- > From: Jakub Kicinski <jakub.kicinski@xxxxxxxxxxxxx> > Sent: Thursday, November 7, 2019 7:16 PM > To: Parav Pandit <parav@xxxxxxxxxxxx> > Cc: alex.williamson@xxxxxxxxxx; davem@xxxxxxxxxxxxx; > kvm@xxxxxxxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; Saeed Mahameed > <saeedm@xxxxxxxxxxxx>; kwankhede@xxxxxxxxxx; leon@xxxxxxxxxx; > cohuck@xxxxxxxxxx; Jiri Pirko <jiri@xxxxxxxxxxxx>; linux- > rdma@xxxxxxxxxxxxxxx; Or Gerlitz <gerlitz.or@xxxxxxxxx> > Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support > > On Thu, 7 Nov 2019 20:52:29 +0000, Parav Pandit wrote: > > > On Thu, 7 Nov 2019 10:04:48 -0600, Parav Pandit wrote: > > > > Mellanox sub function capability allows users to create several > > > > hundreds of networking and/or rdma devices without depending on > > > > PCI SR- > > > IOV support. > > > > > > You call the new port type "sub function" but the devlink port flavour is > mdev. > > > > > Sub function is the internal driver structure. The abstract entity at user and > stack level is mdev. > > Hence the port flavour is mdev. > > FWIW I agree mdev as flavour seems like the right choice. > Ok. > > > As I'm sure you remember you nacked my patches exposing NFP's PCI > > > sub functions which are just regions of the BAR without any mdev > > > capability. Am I in the clear to repost those now? Jiri? > > > > > For sure I didn't nack it. :-) > > Well, maybe the word "nack" wasn't exactly used :) > > > What I remember discussing offline/mailing list is > > (a) exposing mdev/sub fuctions as devlink sub ports is not so good > > abstraction > > (b) user creating/deleting eswitch sub ports would be hard to fit in > > the whole usage model > > Okay, so I can repost the "basic" sub functions? > I think so. Would you like post on top of this series as port flavour etc would come by default? Also there is vfio/mdev dependency exist in this series... > > > > Overview: > > > > --------- > > > > Mellanox ConnectX sub functions are exposed to user as a mediated > > > > device (mdev) [2] as discussed in RFC [3] and further during > > > > netdevconf0x13 at [4]. > > > > > > > > mlx5 mediated device (mdev) enables users to create multiple > > > > netdevices and/or RDMA devices from single PCI function. > > > > > > > > Each mdev maps to a mlx5 sub function. > > > > mlx5 sub function is similar to PCI VF. However it doesn't have > > > > its own PCI function and MSI-X vectors. > > > > > > > > mlx5 mdevs share common PCI resources such as PCI BAR region, > > > > MSI-X interrupts. > > > > > > > > Each mdev has its own window in the PCI BAR region, which is > > > > accessible only to that mdev and applications using it. > > > > > > > > Each mlx5 sub function has its own resource namespace for RDMA > resources. > > > > > > > > mdevs are supported when eswitch mode of the devlink instance is > > > > in switchdev mode described in devlink documentation [5]. > > > > > > So presumably the mdevs don't spawn their own devlink instance > > > today, but once mapped via VIRTIO to a VM they will create one? > > > > > mdev doesn't spawn the devlink instance today when mdev is created by > user, like PCI. > > When PCI bus driver enumerates and creates PCI device, there isn't a > devlink instance for it. > > > > But, mdev's devlink instance is created when mlx5_core driver binds to the > mdev device. > > (again similar to PCI, when mlx5_core driver binds to PCI, its devlink > instance is created ). > > > > I should have put the example in patch-15 which creates/deletes devlink > instance of mdev. > > I will revise the commit log of patch-15 to include that. > > Good point. > > Thanks. > > > > It could be useful to specify. > > > > > Yes, its certainly useful. I missed to put the example in commit log of > patch-15. > > > > > > Network side: > > > > - By default the netdevice and the rdma device of mlx5 mdev cannot > > > > send or receive any packets over the network or to any other mlx5 > mdev. > > > > > > Does this mean the frames don't fall back to the repr by default? > > Probably I wasn't clear. > > What I wanted to say is, that frames transmitted by mdev's netdevice and > rdma devices don't go to network. > > These frames goes to representor device. > > User must configure representor to send/receive/steer traffic to mdev. > > 👍
