On 29/02/2024 16:10, Jason Gunthorpe wrote:
On Wed, Feb 28, 2024 at 12:07:06PM -0700, Alex Williamson wrote:
On Mon, 26 Feb 2024 15:24:58 -0700
Alex Williamson <alex.williamson@xxxxxxxxxx> wrote:
On Mon, 26 Feb 2024 15:49:52 -0400
Jason Gunthorpe <jgg@xxxxxxxxxx> wrote:
On Mon, Feb 26, 2024 at 12:41:07PM -0700, Alex Williamson wrote:
On Mon, 26 Feb 2024 15:12:20 -0400
libvirt recently implemented support for managed="yes" with variant
drivers where it will find the best "vfio_pci" driver for a device
using an algorithm like Max suggested, but in practice it's not clear
how useful that will be considering devices likes CX7 require
configuration before binding to the variant driver. libvirt has no
hooks to specify or perform configuration at that point.
I don't think this is fully accurate (or at least not what was
intended), the VFIO device can be configured any time up until the VM
mlx5 driver reaches the device startup.
Is something preventing this? Did we accidentally cache the migratable
flag in vfio or something??
I don't think so, I think this was just the policy we had decided
relative to profiling VFs when they're created rather than providing a
means to do it though a common vfio variant driver interface[1].
Turns out that yes, migration support needs to be established at probe
time. vfio_pci_core_register_device() expects migration_flags,
mig_ops, and log_ops to all be established by this point, which for
mlx5-vfio-pci occurs when the .init function calls
mlx5vf_cmd_set_migratable().
This is unfortunate, we should look at trying to accomodate this,
IMHO. Yishai?
I'm not sure what is the alternative here.
Moving to the open() might be too late if the guest driver will be
active already, this might prevent changing/setting some caps/profiling,
by that time.
In addition, as Alex wrote, today upon probe() each driver calls
vfio_pci_core_register_device() and by that time the 'mig/log' ops are
checked.
Making a change here, I believe, requires a good justification / use
case and a working alternative and design.
Note:
The above is true for other migration drivers around which upon probe
should supply their 'mig/log' ops.
That also makes me wonder what happens if migration support is disabled
via devlink after binding the VF to mlx5-vfio-pci. Arguably this could
be considered user error, but what's the failure mode and support
implication? Thanks,
I think tThe FW will start failing the migration commands.
Right, please see my previous answer here, I supplied some further details.
Yishai
