On Thu, Aug 16, 2018 at 06:20:29PM -0400, Laine Stump wrote: > Summary of the problem: > > 1) We want to persuade libvirt+QEMU users to move away from the i440fx > machinetype in favor of Q35. (NB: Someday this *might* lead to the > ability to deprecate and even remove the 440fx machinetype, but even if > that were to happen, it would be a *very long* time from now, so this > discussion is *not* about that!) There are plenty of OS that will never support Q35 and are still interesting to use under Q35. The set which could use Q35, but lack virtio1.0 is fairly small. So removal of i440fx is really only something for downstream KVM vendors to consider. Those vendors only care about modern OS, but upstream is much more open minded about what QEMU is used for, so I see it probably living forever in upstream, or at least long enough that current maintaniers will be retired ;-P. > 2) When Q35 machinetype is used, libvirt assigns virtio devices to a > slot on a PCI Express controller (because why have modern PCIe > controllers/slots available but force everything onto clunky old legacy > controllers?). > > 3) When a virtio device is plugged into an Express controller, QEMU > disables the device's IO port space, and it is put into "modern-only" > mode (this is done to avoid a rapid exhaustion of limited IO port space). > > 4) modern-only virtio devices won't work with a legacy (virtio-0.9-only) > guest driver, because virtio-0.9 requires IO port space. > > 5) Some guest OSes that we still want to support (and which would > otherwise work okay on a Q35 virtual machine) have virtio drivers too > old to support virtio-1.0 (CentOS6 and RHEL6 are examples of such OSes), > but due to the chain of reasons listed above, the "standard" config for > a Q35 guest generated by libvirt doesn't support virtio-0.9, hence > doesn't support these guest OSes. Note when talking about "support" you're really saying it from the downstream vendor, specifically RHEL, POV. From upstream or Fedora POV essentially all x86 OS ever made are in scope for running under QEMU if suitable virtual hardware models have been provided. QEMU doesn't maintain any whitelist of "supported" OS that differs from what is technically capable of being run, in the way downstream vendors do. > And here's a list of possible solutions to this problem (note that > "consumers" means management applications such as OpenStack, oVirt, > virt-manager, virt-install, gnome-boxes, etc. In all cases, it's assumed > that the consumer's decision on the action to take will be based on > information from libosinfo). For completeness, I've included even the > possibilities that have been rejected, along with a brief synopsis of > (at least part of) the reason for rejection: > > (1) Add some way libvirt consumers can ask libvirt to place > virtio devices on a legacy pci slot instead of pcie when > the machinetype is q35 (qemu sets virtio devices in legacy > PCI slots to transitional mode, so io port space is enabled > and virtio-0.0 drivers will work). > > This has been proposed on libvir-list, but rejected. Here is > the most elquently stated reasoning for the rejection I could > find (with thanks to Dan Berrange): > > The domain XML is a way to express the configuration > of the guest virtual machine. What we're talking about > here is a policy tunable for an internal libvirt QEMU > driver algorithm, as so does not belong anywhere in the > domain XML. Indeed, that's a guiding principal in general, not just for this PCI question. > (2) Add full-blown pci enumeration support to all libvirt consumers > (i.e. they will need to build a model of the PCI bus topology > of each guest, and keep track of which addresses are in use). > They can then manually place virtio devices on legacy pci slots > (again, triggering transitional mode) when the intended guest > OS doesn't support virtio-0.9. > > (This is seen as requiring too much duplicated effort for > development and support/maintenance, since up until now libvirt > has been the single point of action for PCI address assignment > (well, QEMU can do it too, but for > 10 years libvirt has > *always* provided full PCI addresses for all devices) It really depends on the scope of the mgmt app - at some point the mgmt apps needs to take charge to some degree if it has particular ideas about how a machine should look. Libvirt's placement strategy is a good default for 95% of use cases, but it'll never be 100%. An example is setting up a particular PCI topology that is guest NUMA node aware, using expander buses. So some apps might take this option, but in the common case it is undesirable. > (3) Add virtio-1.0 support to all guest OSes. If this is done, > existing libvirt configs will work. > > (Aside from the difficulty of backporting, and the fact that > there are going to be some OSes that don't get it *at all*, > there will always be older releases that haven't gotten the > backport. So this isn't a complete solution). Yep, there will always be guest OS that don't support 1.0. So that's only a solution if the person who cares about Q35 support also controls the guest OS in question. > (4) Consumers can continue using the 440fx machinetype for guest > OSes that don't support virtio-0.9 > > (This would work, but perpetuates use of the 440fx > machinetype, and all for just this one reason (at least in > the case of CentOS6/RHEL6, which otherwise work just fine with > Q35)). >From an usptream POV this is always going to be the case. This is really only an undesirable thing for downstream who are trying to artificially restrict what QEMU features users have available to them. > (5) Introduce virtio-0.9, virtio-1.0 models in libvirt > which are explicitly legacy-only and modern-only. > QEMU doesn't need to change, as libvirt can simply set > the right params on existing QEMU models to force the > behavior. > > (NB: it's unclear to me whether virtio-0.9 simply won't > work without forcing the device to be on a legacy PCI > slot, or if that's just "a very bad idea" because it > will mean that the device uses up extra io port space) > As a starter for continuing the discussion, it seems to me that for > option (5): > > a) we don't really need the virtio-1.0 model, since that's what you > currently get anyway when you ask for "virtio" on Q35 (and on 440fx, > "virtio" gives you transitional, which works for everybody). At some point we might have a virtio-2.0 and find ourselves in a similar problem again. IMHO it is preferrable to have both explicit versioned models, and discourage use of the magical 'virtio' model from mgmt apps. Use libosinfo to identify which virtio model is supported for the OS in question and use that explicitly. Only use the magical 'virtio' model if there's no information about what version the OS supports. > b) Rather than a "legacy-only" model for virtio-0.9, it would be more > useful to have "transitional". This way the config would work for older > OSes that don't support virtio-1.0, and when/if the OS was upgraded such > that it supported virtio-1.0, that would be automatically used without > needing to change the config. I don't think the case of OS suddenly gaining support for 1.0 in an update is frequent enough to be worth worrying about. > c) Even if it's possible to force a device on an Express slot into > transitional mode, this is extremely wasteful of io port space, so > libvirt should consider virtio-0.9 devices to be legacy PCI, and thus > plug them into legacy PCI slots. And once we're doing this, it's > unnecessary to add any extra option to the qemu commandline to force > legacy support (i.e. transitional mode), as that is what QEMU already > does when the device is connected to a legacy PCI slot. Yes, it should plug them into legacy PCI slots by default, but if a mgmt app has done explicit placement itself, it should honour that even if it means wasting IO space. > So making the naive assumption that we agree on implementing option (5) > and there are no objections to my points a-c (Hah! As if!), how does > this sound as a plan: > > > A) libosinfo starts telling consumers that the preferred virtio device > model for the relevant OSes is "virtio-0.9", and leaves the > recommendation for other OSes as "virtio". Libosinfo already uses 'virtio' as the prefix identifying virtio-0.9 support (the old PCI product IDs), and 'virtio-1.0' as the prefix for identifying virtio-1.0 support (the new PCI product IDs). That these don't match libvirt model names doesn't matter. > B) libvirt adds a "virtio-0.9" model for all virtio devices that > actually have virtio-0.9 support (a couple of devices never existed > prior to virtio-1.0 (rng and ???) so virtio-0.9 would be nonsensical for > them). > > C) inside libvirt, the implementation of the "virtio-0.9" model is > identical to "virtio", except that the VIR_PCI_CONNECT_TYPE flags for > these devices contain VIR_PCI_CONNECT_TYPE_PCI rather than > VIR_PCI_CONNECT_TYPE_PCIE, resulting in those devices being assigned to > a legacy PCI slot, and thus they would be transitional mode by default. For 'virtio-0.9' libvirt should set "disable-modern=yes" in QEMU args For 'virtio-1.0' libvirt should set "disable-legacy=yes" in QEMU args Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
