On Mon, Feb 15, 2021 at 03:01:06PM -0600, Bjorn Helgaas wrote:
> On Tue, Feb 09, 2021 at 03:34:42PM +0200, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@xxxxxxxxxx>
> >
> > Extend PCI sysfs interface with a new callback that allows configuration
> > of the number of MSI-X vectors for specific SR-IOV VF. This is needed
> > to optimize the performance of VFs devices by allocating the number of
> > vectors based on the administrator knowledge of the intended use of the VF.
> >
> > This function is applicable for SR-IOV VF because such devices allocate
> > their MSI-X table before they will run on the VMs and HW can't guess the
> > right number of vectors, so some devices allocate them statically and equally.
>
> This commit log should be clear that this functionality is motivated
> by *mlx5* behavior. The description above makes it sound like this is
> generic PCI spec behavior, and it is not.
>
> It may be a reasonable design that conforms to the spec, and we hope
> the model will be usable by other designs, but it is not required by
> the spec and AFAIK there is nothing in the spec you can point to as
> background for this.
>
> So don't *remove* the text you have above, but please *add* some
> preceding background information about how mlx5 works.
>
> > 1) The newly added /sys/bus/pci/devices/.../sriov_vf_msix_count
> > file will be seen for the VFs and it is writable as long as a driver is not
> > bound to the VF.
>
> This adds /sys/bus/pci/devices/.../sriov_vf_msix_count for VF
> devices and is writable ...
>
> > The values accepted are:
> > * > 0 - this will be number reported by the Table Size in the VF's MSI-X Message
> > Control register
> > * < 0 - not valid
> > * = 0 - will reset to the device default value
>
> = 0 - will reset to a device-specific default value
>
> > 2) In order to make management easy, provide new read-only sysfs file that
> > returns a total number of possible to configure MSI-X vectors.
>
> For PF devices, this adds a read-only
> /sys/bus/pci/devices/.../sriov_vf_total_msix file that contains the
> total number of MSI-X vectors available for distribution among VFs.
>
> Just as in sysfs-bus-pci, this file should be listed first, because
> you must read it before you can use vf_msix_count.
No problem, I'll change, just remember that we are talking about commit
message because in Documentation file, the order is exactly as you request.
>
> > cat /sys/bus/pci/devices/.../sriov_vf_total_msix
> > = 0 - feature is not supported
> > > 0 - total number of MSI-X vectors available for distribution among the VFs
> >
> > Signed-off-by: Leon Romanovsky <leonro@xxxxxxxxxx>
> > ---
> > Documentation/ABI/testing/sysfs-bus-pci | 28 +++++
> > drivers/pci/iov.c | 153 ++++++++++++++++++++++++
> > include/linux/pci.h | 12 ++
> > 3 files changed, 193 insertions(+)
<...>
> > + */
> > +int pci_enable_vf_overlay(struct pci_dev *dev)
> > +{
> > + struct pci_dev *virtfn;
> > + int id, ret;
> > +
> > + if (!dev->is_physfn || !dev->sriov->num_VFs)
> > + return 0;
> > +
> > + ret = sysfs_create_files(&dev->dev.kobj, sriov_pf_dev_attrs);
>
> But I still don't like the fact that we're calling
> sysfs_create_files() and sysfs_remove_files() directly. It makes
> complication and opportunities for errors.
It is not different from any other code that we have in the kernel.
Let's be concrete, can you point to the errors in this code that I
should fix?
>
> I don't see the advantage of creating these files only when the PF
> driver supports this. The management tools have to deal with
> sriov_vf_total_msix == 0 and sriov_vf_msix_count == 0 anyway.
> Having the sysfs files not be present at all might be slightly
> prettier to the person running "ls", but I'm not sure the code
> complication is worth that.
It is more than "ls", right now sriov_numvfs is visible without relation
to the driver, even if driver doesn't implement ".sriov_configure", which
IMHO bad. We didn't want to repeat.
Right now, we have many devices that supports SR-IOV, but small amount
of them are capable to rewrite their VF MSI-X table siz. We don't want
"to punish" and clatter their sysfs.
>
> I see a hint that Alex might have requested this "only visible when PF
> driver supports it" functionality, but I don't see that email on
> linux-pci, so I missed the background.
First version of this patch had static files solution.
https://lore.kernel.org/linux-pci/20210103082440.34994-2-leon@xxxxxxxxxx/#Z30drivers:pci:iov.c
Thanks
