On Mon, Mar 01, 2021 at 09:37:17AM +0100, Greg Kroah-Hartman wrote: > On Mon, Mar 01, 2021 at 10:32:09AM +0200, Leon Romanovsky wrote: > > On Mon, Mar 01, 2021 at 09:14:42AM +0100, Greg Kroah-Hartman wrote: > > > On Mon, Mar 01, 2021 at 09:55:21AM +0200, Leon Romanovsky wrote: > > > > From: Leon Romanovsky <leonro@xxxxxxxxxx> > > > > > > > > A typical cloud provider SR-IOV use case is to create many VFs for use by > > > > guest VMs. The VFs may not be assigned to a VM until a customer requests a > > > > VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors > > > > proportional to the number of CPUs in the VM, but there is no standard way > > > > to change the number of MSI-X vectors supported by a VF. > > > > > > > > Some Mellanox ConnectX devices support dynamic assignment of MSI-X vectors > > > > to SR-IOV VFs. This can be done by the PF driver after VFs are enabled, > > > > and it can be done without affecting VFs that are already in use. The > > > > hardware supports a limited pool of MSI-X vectors that can be assigned to > > > > the PF or to individual VFs. This is device-specific behavior that > > > > requires support in the PF driver. > > > > > > > > Add a read-only "sriov_vf_total_msix" sysfs file for the PF and a writable > > > > "sriov_vf_msix_count" file for each VF. Management software may use these > > > > to learn how many MSI-X vectors are available and to dynamically assign > > > > them to VFs before the VFs are passed through to a VM. > > > > > > > > If the PF driver implements the ->sriov_get_vf_total_msix() callback, > > > > "sriov_vf_total_msix" contains the total number of MSI-X vectors available > > > > for distribution among VFs. > > > > > > > > If no driver is bound to the VF, writing "N" to "sriov_vf_msix_count" uses > > > > the PF driver ->sriov_set_msix_vec_count() callback to assign "N" MSI-X > > > > vectors to the VF. When a VF driver subsequently reads the MSI-X Message > > > > Control register, it will see the new Table Size "N". > > > > > > > > Signed-off-by: Leon Romanovsky <leonro@xxxxxxxxxx> > > > > --- > > > > Documentation/ABI/testing/sysfs-bus-pci | 29 +++++++ > > > > drivers/pci/iov.c | 102 ++++++++++++++++++++++-- > > > > drivers/pci/pci-sysfs.c | 3 +- > > > > drivers/pci/pci.h | 3 +- > > > > include/linux/pci.h | 8 ++ > > > > 5 files changed, 137 insertions(+), 8 deletions(-) > > > > > > > > diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci > > > > index 25c9c39770c6..ebabd0d2ae88 100644 > > > > --- a/Documentation/ABI/testing/sysfs-bus-pci > > > > +++ b/Documentation/ABI/testing/sysfs-bus-pci > > > > @@ -375,3 +375,32 @@ Description: > > > > The value comes from the PCI kernel device state and can be one > > > > of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold". > > > > The file is read only. > > > > + > > > > +What: /sys/bus/pci/devices/.../sriov_vf_total_msix > > > > +Date: January 2021 > > > > +Contact: Leon Romanovsky <leonro@xxxxxxxxxx> > > > > +Description: > > > > + This file is associated with a SR-IOV PF. It contains the > > > > + total number of MSI-X vectors available for assignment to > > > > + all VFs associated with PF. It will zero if the device > > > > > > "The value will be zero if the device..." > > > > Thanks, will fix when apply or resend if more fixes will be needed. > > > > > > > > And definition of "VF" and PF" are where in this file? > > > > They come from the PCI spec. It is part of SR-IOV lingo. > > Yes, and the world does not have access to the PCI spec, so please > provide a hint as to what they are for those of us without access to > such things. No problem, I will replace PF to be "physical function" and VF to be "virtual function". Thanks > > thanks, > > greg k-h
