On Mon, Aug 05, 2024 at 05:35:17AM +0000, Tian, Kevin wrote: > Okay. With that I edited my earlier reply a bit by removing the note > of cmdline option, adding DVSEC possibility, and making it clear that > the PASID option is in vIOMMU: > > " > Overall this sounds a feasible path to move forward - starting with > the VMM to find the gap automatically if PASID is opted in vIOMMU. > Devices with hidden registers may fail. Devices with volatile > config space due to FW upgrade or cross vendors may fail to migrate. > Then evolving it to the file-based scheme, and there is time to discuss > any intermediate improvement (fixed quirks, DVSEC, etc.) in between. > " > > Jason, your thoughts? This thread is big and I've read it quickly, but I could support the above summary. For migration, where we are today, it is completely up to the device and it's FW to present a new config space that is stable across migration. In practice this means the device needs to have pre-set the PF to whatever config state it needs during FLR, and refuse to migrate if things are not correct. Moving away from that limitation toward a VMM created stable config space is definitely a long term interest. I understand other VMM's are already doing things like this. It seems we need more time to bake on this long term direction. I suggested a text file as something very general, but we can do other things too. What I see as the general industry direction is toward a very perspective vPCI function, where you might have v1, v2, etc of a device that present different config space/behavior/etc. I expect standards bodies are going to define reference vPCI functions for their standards along these lines. This would specify all the MMIO memory layout and every bit of config space right down to each offset and bit. This is the sort of restriction that will be needed to allow more generic live migration between different devices and different FW. There is a pretty clear role here for the VMM to synthesize a highly perscribed config space. How we get there and how the libvirt stack should support this, I don't know. Jason
