On Mon, Sep 18, 2023 at 08:43:21PM +0200, Thomas Gleixner wrote: > On Mon, Sep 18 2023 at 11:17, Jason Gunthorpe wrote: > > On Thu, Sep 14, 2023 at 12:14:06PM -0700, Shannon Nelson wrote: > >> The new MSI dynamic allocation machinery is great for making the irq > >> management more flexible. It includes caching information about the > >> MSI domain which gets reused on each new open of a VFIO fd. However, > >> this causes an issue when the underlying hardware has flexible MSI-x > >> configurations, as a changed configuration doesn't get seen between > >> new opens, and is only refreshed between PCI unbind/bind cycles. > >> > >> In our device we can change the per-VF MSI-x resource allocation > >> without the need for rebooting or function reset. For example, > >> > >> 1. Initial power up and kernel boot: > >> # lspci -s 2e:00.1 -vv | grep MSI-X > >> Capabilities: [a0] MSI-X: Enable+ Count=8 Masked- > >> > >> 2. Device VF configuration change happens with no reset > > > > Is this an out of tree driver problem? > > > > The intree way to alter the MSI configuration is via > > sriov_set_msix_vec_count, and there is only one in-tree driver that > > uses it right now. > > Right, but that only addresses the driver specific issues. Sort of.. sriov_vf_msix_count_store() is intended to be the entry point for this and if the kernel grows places that cache the value or something then this function should flush those caches too. I suppose flushing happens implicitly because Shannon reports that things work fine if the driver is rebound. Since sriov_vf_msix_count_store() ensures there is no driver bound before proceeding it probe/unprobe must be flushing out everything? Jason
