Bjorn, Can you please help us progress with the series? It needs to go from mlx5-next to net-next and maybe to rdma-next too and we need enough time to keep the series for automatic testing in all those branches. The code was posted on -rc1 [1] while we are in -rc4 already. [1] https://lore.kernel.org/linux-pci/20210301075524.441609-1-leon@xxxxxxxxxx/ Thanks On Sat, Mar 20, 2021 at 11:13:44AM +0200, Leon Romanovsky wrote: > Gentle reminder > > Thanks > > On Sun, Mar 14, 2021 at 02:42:52PM +0200, Leon Romanovsky wrote: > > --------------------------------------------------------------------------------- > > Changelog > > v8: > > * Added "physical/virtual function" words near PF and VF acronyms. > > v7: https://lore.kernel.org/linux-pci/20210301075524.441609-1-leon@xxxxxxxxxx > > * Rebase on top v5.12-rc1 > > * More english fixes > > * Returned to static sysfs creation model as was implemented in v0/v1. > > v6: https://lore.kernel.org/linux-pci/20210209133445.700225-1-leon@xxxxxxxxxx > > * Patch 1: > > * English fixes > > * Moved pci_vf_set_msix_vec_count() from msi.c to iov.c > > * Embedded pci_vf_set_msix_vec_count() into sriov_vf_msix_count_store > > * Deleted sriov_vf_msix_count_show > > * Deleted vfs_overlay folder > > * Renamed functions *_vfs_overlay_* to be *_vf_overlay_* > > * Deleted is_supported and attribute_group because it confused people more than > > it gave advantage. > > * Changed vf_total_msix to be callback > > * Patch 3: > > * Fixed english as suggested by Bjorn > > * Added more explanations to the commit message > > * Patch 4: > > * Protected enable/disable with capability check > > v5: https://lore.kernel.org/linux-pci/20210126085730.1165673-1-leon@xxxxxxxxxx > > * Patch 1: > > * Added forgotten "inline" keyword when declaring empty functions. > > v4: https://lore.kernel.org/linux-pci/20210124131119.558563-1-leon@xxxxxxxxxx > > * Used sysfs_emit() instead of sprintf() in new sysfs entries. > > * Changed EXPORT_SYMBOL to be EXPORT_SYMBOL_GPL for pci_iov_virtfn_devfn(). > > * Rewrote sysfs registration code to be driven by PF that wants to enable VF > > overlay instead of creating to all SR-IOV devices. > > * Grouped all such functionality under new "vfs_overlay" folder. > > * Combined two PCI patches into one. > > v3: https://lore.kernel.org/linux-pci/20210117081548.1278992-1-leon@xxxxxxxxxx > > * Renamed pci_set_msix_vec_count to be pci_vf_set_msix_vec_count. > > * Added VF msix_cap check to hide sysfs entry if device doesn't support msix. > > * Changed "-" to be ":" in the mlx5 patch to silence CI warnings about missing > > kdoc description. > > * Split differently error print in mlx5 driver to avoid checkpatch warning. > > v2: https://lore.kernel.org/linux-pci/20210114103140.866141-1-leon@xxxxxxxxxx > > * Patch 1: > > * Renamed vf_msix_vec sysfs knob to be sriov_vf_msix_count > > * Added PF and VF device locks during set MSI-X call to protect from parallel > > driver bind/unbind operations. > > * Removed extra checks when reading sriov_vf_msix, because users will > > be able to distinguish between supported/not supported by looking on > > sriov_vf_total_msix count. > > * Changed all occurrences of "numb" to be "count" > > * Changed returned error from EOPNOTSUPP to be EBUSY if user tries to set > > MSI-X count after driver already bound to the VF. > > * Added extra comment in pci_set_msix_vec_count() to emphasize that driver > > should not be bound. > > * Patch 2: > > * Changed vf_total_msix from int to be u32 and updated function signatures > > accordingly. > > * Improved patch title > > v1: https://lore.kernel.org/linux-pci/20210110150727.1965295-1-leon@xxxxxxxxxx > > * Improved wording and commit messages of first PCI patch > > * Added extra PCI patch to provide total number of MSI-X vectors > > * Prohibited read of vf_msix_vec sysfs file if driver doesn't support write > > * Removed extra function definition in pci.h > > v0: https://lore.kernel.org/linux-pci/20210103082440.34994-1-leon@xxxxxxxxxx > > > > -------------------------------------------------------------------- > > Hi, > > > > The number of MSI-X vectors is PCI property visible through lspci, that > > field is read-only and configured by the device. > > > > The static assignment of an amount of MSI-X vectors doesn't allow utilize > > the newly created VF because it is not known to the device the future load > > and configuration where that VF will be used. > > > > The VFs are created on the hypervisor and forwarded to the VMs that have > > different properties (for example number of CPUs). > > > > To overcome the inefficiency in the spread of such MSI-X vectors, we > > allow the kernel to instruct the device with the needed number of such > > vectors, before VF is initialized and bounded to the driver. > > > > Before this series: > > [root@server ~]# lspci -vs 0000:08:00.2 > > 08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] > > .... > > Capabilities: [9c] MSI-X: Enable- Count=12 Masked- > > > > Configuration script: > > 1. Start fresh > > echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs > > modprobe -q -r mlx5_ib mlx5_core > > 2. Ensure that driver doesn't run and it is safe to change MSI-X > > echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe > > 3. Load driver for the PF > > modprobe mlx5_core > > 4. Configure one of the VFs with new number > > echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs > > echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/sriov_vf_msix_count > > > > After this series: > > [root@server ~]# lspci -vs 0000:08:00.2 > > 08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] > > .... > > Capabilities: [9c] MSI-X: Enable- Count=21 Masked- > > > > Thanks > > > > Leon Romanovsky (4): > > PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs > > net/mlx5: Add dynamic MSI-X capabilities bits > > net/mlx5: Dynamically assign MSI-X vectors count > > net/mlx5: Implement sriov_get_vf_total_msix/count() callbacks > > > > Documentation/ABI/testing/sysfs-bus-pci | 29 +++++ > > .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ > > .../ethernet/mellanox/mlx5/core/mlx5_core.h | 12 +++ > > .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 73 +++++++++++++ > > .../net/ethernet/mellanox/mlx5/core/sriov.c | 48 ++++++++- > > drivers/pci/iov.c | 102 ++++++++++++++++-- > > drivers/pci/pci-sysfs.c | 3 +- > > drivers/pci/pci.h | 3 +- > > include/linux/mlx5/mlx5_ifc.h | 11 +- > > include/linux/pci.h | 8 ++ > > 10 files changed, 284 insertions(+), 11 deletions(-) > > > > -- > > 2.30.2 > >
