On Wed, Dec 07, 2022 at 12:59:00PM +0200, Max Gurtovoy wrote: > Why is it preferred that the migration SW will talk directly to the PF and > not via VFIO interface ? It should never talk directly to any hardware, but through a kernel interface, and that's probably vfio. But that interface needs to centered around the controlling function for all the reasons I've written down multiple times now. > It's just an implementation detail. No, it's not. While you could come up with awkward ways to map how the hardware interface must work to a completely contrary kernel interface that's just going to create the need for lots of boilerplate code _and_ confuses users. The function that is beeing migrated can fundamentally not be in control of itself. Any interface that pretends it is broken and a long term nightmare for users and implementers. > I feel like it's even sounds more reasonable to have a common API like we > have today to save_state/resume_state/quiesce_device/freeze_device and each > device implementation will translate this functionality to its own SPEC. Absolutely. > If I understand your direction is to have QEMU code to talk to > nvmecli/new_mlx5cli/my_device_cli to do that and I'm not sure it's needed. No. > The controlled device is not aware of any of the migration process. Only > the migration SW, system admin and controlling device. Exactly. > So in the source: > > 1. We enable SRIOV on the NVMe driver Again. Nothing in live migration is tied to SR-IOV at all. SR-IOV is just one way to get multiple functions. > 2. We list all the secondary controllers: nvme1, nvme2, nvme3 > > 3. We allow migrating nvme1, nvme2, nvme3 - now these VFs are migratable > (controlling to controlled). > > 4. We bind nvme1, nvme2, nvme3 to VFIO NVMe driver > > 5. We pass these functions to VM And you need to pass the controlling function (or rather a handle for it), because there is absolutely no sane way to discover that from the controlled function as it can't have that information by the fact that it is beeing passed to unprivilged VMs.
