On Wed, 2023-05-31 at 12:35 +0100, Catalin Marinas wrote: > There were several off-list discussions, I'm trying to summarise my > understanding here. This series aims to relax the VFIO mapping to > cacheable and have KVM map it into the guest with the same attributes. > Somewhat related past threads also tried to relax the KVM device > pass-through mapping from Device_nGnRnE (pgprot_noncached) to Normal_NC > (pgprot_writecombine). Those were initially using the PCIe prefetchable > BAR attribute but that's not a reliable means to infer whether Normal vs > Device is safe. Anyway, I think we'd need to unify these threads and > come up with some common handling that can cater for various attributes > required by devices/drivers. Therefore replying in this thread. So picking up on this as I was just trying to start a separate discussion on the subject for write combine :-) In this case, not so much for KVM as much as for VFIO to userspace though. The rough idea is that the "userspace driver" (ie DPDK or equivalent) for the device is the one to "know" wether a BAR or portion of a BAR can/should be mapped write-combine, and is expected to also "know" what to do to enforce ordering when necessary. So the userspace component needs to be responsible for selecting the mapping, the same way using the PCI sysfs resource files today allows to do that by selecting the _wc variant. I posted a separate message that Lorenzo CCed back to some of you, but let's recap here to keep the discussion localized. I don't know how much of this makes sense for KVM, but I think what we really want is for userspace to be able to specify some "attributes" (which we can initially limit to writecombine, full cachability probably requires a device specific kernel driver providing adequate authority, separate discussion in any case), for all or a portion of a BAR mapping. The easy way is an ioctl to affect the attributes of the next mmap but it's a rather gross interface. A better approach (still requires some coordination but not nearly as bad) would be to have an ioctl to create "subregions", ie, dynamically add new "struct vfio_pci_region" (using the existing dynamic index API), which are children of existing regions (including real BARs) and provide different attributes, which mmap can then honor. This is particularly suited for the case (which used to exist, I don't know if it still does) where the buffer that wants write combining reside in the same BAR as registers that otherwise don't. A simpler compromise if that latter case is deemed irrelevant would be an ioctl to selectively set a region index (including BARs) to be WC prior to mmap. I don't know if that fits in the ideas you have for KVM, I think it could by having the userspace component require mappings using a "special" attribute which we could define as being the most relaxed allowed to pass to a VM, which can then be architecture defined. The guest can then enforce specifics. Does this make sense ? Cheers Ben.
