On Tue, Dec 06, 2022 at 03:31:26PM +0100, Christoph Hellwig wrote: > On Tue, Dec 06, 2022 at 10:20:26AM -0400, Jason Gunthorpe wrote: > > In the VFIO restore model there is no "live OS" on resume. The > > load/resume cycle is as destructive as reset to the vfio device. > > Of course there may be and OS. As soon as the VF is live Linux > will by default bind to it. And that's the big problem here, > the VF should not actually exist or at least not be usable when > such a restore happens - or to say it in NVMe terms, the Secondary > Controller better be in offline state when state is loaded into it. Sadly in Linux we don't have a SRIOV VF lifecycle model that is any use. What we do have is a transfer of control from the normal OS driver (eg nvme) to the VFIO driver. Also, remember, that VFIO only does live migration between VFIO devices. We cannot use live migration and end up with a situation where the normal nvme driver is controlling the VF. The VFIO load model is explicitly destructive. We replace the current VF with the loading VF. Both the VFIO variant driver and the VFIO userspace issuing the load have to be aware of this and understand that the whole device will change. >From an implementation perspective, I would expect the nvme varient driver to either place the nvme device in the correct state during load, or refuse to execute load if it is in the wrong state. To be compatible with what qemu is doing the "right state" should be entered by completing function level reset of the VF. The Linux/qemu parts are still being finalized, so if you see something that could be changed to better match nvme it would be a great time to understand that. Jason
