> -----Original Message----- > From: Jiri Pirko <jiri@xxxxxxxxxxx> > Sent: Friday, November 8, 2019 6:13 AM > To: Jakub Kicinski <jakub.kicinski@xxxxxxxxxxxxx> > Cc: Parav Pandit <parav@xxxxxxxxxxxx>; alex.williamson@xxxxxxxxxx; > davem@xxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; > Saeed Mahameed <saeedm@xxxxxxxxxxxx>; kwankhede@xxxxxxxxxx; > leon@xxxxxxxxxx; cohuck@xxxxxxxxxx; Jiri Pirko <jiri@xxxxxxxxxxxx>; linux- > rdma@xxxxxxxxxxxxxxx; Or Gerlitz <gerlitz.or@xxxxxxxxx> > Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support > > Thu, Nov 07, 2019 at 09:32:34PM CET, jakub.kicinski@xxxxxxxxxxxxx wrote: > >On Thu, 7 Nov 2019 10:04:48 -0600, Parav Pandit wrote: > >> Mellanox sub function capability allows users to create several > >> hundreds of networking and/or rdma devices without depending on PCI > SR-IOV support. > > > >You call the new port type "sub function" but the devlink port flavour > >is mdev. > > > >As I'm sure you remember you nacked my patches exposing NFP's PCI sub > >functions which are just regions of the BAR without any mdev > >capability. Am I in the clear to repost those now? Jiri? > > Well question is, if it makes sense to have SFs without having them as mdev? > I mean, we discussed the modelling thoroughtly and eventually we realized > that in order to model this correctly, we need SFs on "a bus". > Originally we were thinking about custom bus, but mdev is already there to > handle this. > > Our SFs are also just regions of the BAR, same thing as you have. > > Can't you do the same for nfp SFs? > Then the "mdev" flavour is enough for all. > > > > > >> Overview: > >> --------- > >> Mellanox ConnectX sub functions are exposed to user as a mediated > >> device (mdev) [2] as discussed in RFC [3] and further during > >> netdevconf0x13 at [4]. > >> > >> mlx5 mediated device (mdev) enables users to create multiple > >> netdevices and/or RDMA devices from single PCI function. > >> > >> Each mdev maps to a mlx5 sub function. > >> mlx5 sub function is similar to PCI VF. However it doesn't have its > >> own PCI function and MSI-X vectors. > >> > >> mlx5 mdevs share common PCI resources such as PCI BAR region, MSI-X > >> interrupts. > >> > >> Each mdev has its own window in the PCI BAR region, which is > >> accessible only to that mdev and applications using it. > >> > >> Each mlx5 sub function has its own resource namespace for RDMA > resources. > >> > >> mdevs are supported when eswitch mode of the devlink instance is in > >> switchdev mode described in devlink documentation [5]. > > > >So presumably the mdevs don't spawn their own devlink instance today, > >but once mapped via VIRTIO to a VM they will create one? > > I don't think it is needed for anything. Maybe one day if there is a need to > create devlink instance for VF or SF, we can add it. But currently, I don't see > the need. > > > > > >It could be useful to specify. > > > >> Network side: > >> - By default the netdevice and the rdma device of mlx5 mdev cannot > >> send or receive any packets over the network or to any other mlx5 mdev. > > > >Does this mean the frames don't fall back to the repr by default? > > That would be the sane default. If I up the representor, I should see packets > coming in from SF/VF and I should be able to send packets back. It is similar to VF. I clarified in previous email. Wil update the cover letter to update the description.
