On Fri, Apr 26, 2024 at 02:13:54PM -0600, Alex Williamson wrote: > On Fri, 26 Apr 2024 11:11:17 -0300 > Jason Gunthorpe <jgg@xxxxxxxxxx> wrote: > > > On Wed, Apr 24, 2024 at 02:13:49PM -0600, Alex Williamson wrote: > > > > > This is kind of an absurd example to portray as a ubiquitous problem. > > > Typically the config space layout is a reflection of hardware whether > > > the device supports migration or not. > > > > Er, all our HW has FW constructed config space. It changes with FW > > upgrades. We change it during the life of the product. This has to be > > considered.. > > So as I understand it, the concern is that you have firmware that > supports migration, but it also openly hostile to the fundamental > aspects of exposing a stable device ABI in support of migration. Well, that makes it sound rude, but yes that is part of it. mlx5 is tremendously FW defined. The FW can only cope with migration in some limited cases today. Making that compatability bigger is ongoing work. Config space is one of the areas that has not been addressed. Currently things are such that the FW won't support migration in combinations that have different physical config space - so it is not a problem. But, in principle, it is an issue. AFAIK, the only complete solution is for the hypervisor to fully synthesize a stable config space. So, if we keep this in the kernel then I'd imagine the kernel will need to grow some shared infrastructure to fully synthezise the config space - not text file based, but basically the same as what I described for the VMM. > > But that won't be true if the kernel is making decisions. The config > > space layout depends now on the kernel driver version too. > > But in the cases where we support migration there's a device specific > variant driver that supports that migration. It's the job of that > variant driver to not only export and import the device state, but also > to provide a consistent ABI to the user, which includes the config > space layout. Yes, we could do that, but I'm not sure how it will work in all cases. > I don't understand why we'd say the device programming ABI itself > falls within the purview of the device/variant driver, but PCI > config space is defined by device specific code at a higher level. The "device programming ABI" doesn't contain any *policy*. The layout of the config space is like 50% policy. Especially when we start to talk about standards defined migration. The standards will set the "device programming ABI" and maybe even specify the migration stream. They will not, and arguably can not, specify the config space. Config space layout is substantially policy of the instance type. Even little things like the vendor IDs can be meaningfully replaced in VMs. > Regarding "if we accept that text file configuration should be > something the VMM supports", I'm not on board with this yet, so > applying it to PASID discussion seems premature. Sure, I'm just explaining a way this could all fit together. > We've developed variant drivers specifically to host the device specific > aspects of migration support. The requirement of a consistent config > space layout is a problem that only exists relative to migration. Well, I wouldn't go quite so far. Arguably even non-migritable instance types may want to adjust thier config space. Eg if I'm using a DPU and I get a NVMe/Virtio PCI function I may want to scrub out details from the config space to make it more general. Even without migration. This already happens today in places like VDPA which completely replace the underlying config space in some cases. I see it as a difference from a world of highly constrained "instance types" and a more ad hoc world. > is an issue that I would have considered the responsibility of the > variant driver, which would likely expect a consistent interface from > the hardware/firmware. Why does hostile firmware suddenly make it the > VMM's problem to provide a consistent ABI to the config space of the > device rather than the variant driver? It is not "hostile firmware"! It accepting that a significant aspect of the config layout is actually policy. Plus the standards limitations that mean we can't change the config space on the fly make it pretty much impossible for the device to acutally do anything to help here. Software must fix the config space. > Obviously config maps are something that a VMM could do, but it also > seems to impose a non-trivial burden that every VMM requires an > implementation of a config space map and integration for each device > rather than simply expecting the exposed config space of the device to > be part of the migration ABI. Well, the flip is true to, it is alot of burden on every variant device driver implement and on the kernel in general to manage config space policy on behalf of the VMM. My point is if the VMM is already going to be forced to manage config space policy for other good reasons, are we sure we want to put a bunch of stuff in the kernel that sometimes won't be used? > Also this solution specifically only addresses config space > compatibility without considering the more generic issue that a > variant driver can expose different device personas. A versioned > persona and config space virtualization in the variant driver is a > much more flexible solution. It is addressed, the different personas would have their own text file maps. The target VMM would have to load the right map. Shared common code across all the variant drivers. Jason
