On Wed, Dec 06, 2023 at 11:39:03AM +0000, Marc Zyngier wrote: > On Tue, 05 Dec 2023 18:40:42 +0000, > Catalin Marinas <catalin.marinas@xxxxxxx> wrote: > > On Tue, Dec 05, 2023 at 05:50:27PM +0000, Marc Zyngier wrote: > > > On Tue, 05 Dec 2023 17:33:01 +0000, > > > Catalin Marinas <catalin.marinas@xxxxxxx> wrote: > > > > Ideally we should do this for vfio only but we don't have an easy > > > > way to convey this to KVM. > > > > > > But if we want to limit this to PCIe, we'll have to find out. The > > > initial proposal (a long while ago) had a flag conveying some > > > information, and I'd definitely feel more confident having something > > > like that. > > > > We can add a VM_PCI_IO in the high vma flags to be set by > > vfio_pci_core_mmap(), though it limits it to 64-bit architectures. KVM > > knows this is PCI and relaxes things a bit. It's not generic though if > > we need this later for something else. > > Either that, or something actually describing the attributes that VFIO > wants. > > And I very much want it to be a buy-in behaviour, not something that > automagically happens and changes the default behaviour for everyone > based on some hand-wavy assertions. > > If that means a userspace change, fine by me. The VMM better know what > is happening. Driving the attributes from a single point like the VFIO driver is indeed better. The problem is that write-combining on Arm doesn't come without speculative loads, otherwise we would have solved it by now. I also recall the VFIO maintainer pushing back on relaxing the pgprot_noncached() for the user mapping but I don't remember the reasons. We could do with a pgprot_maybewritecombine() or pgprot_writecombinenospec() (similar to Jason's idea but without changing the semantics of pgprot_device()). For the user mapping on arm64 this would be Device (even _GRE) since it can't disable speculation but stage 2 would leave the decision to the guest since the speculative loads aren't much different from committed loads done wrongly. If we want the VMM to drive this entirely, we could add a new mmap() flag like MAP_WRITECOMBINE or PROT_WRITECOMBINE. They do feel a bit weird but there is precedent with PROT_MTE to describe a memory type. One question is whether the VFIO driver still needs to have the knowledge and sanitise the requests from the VMM within a single BAR. If there are no security implications to such mappings, the VMM can map parts of the BAR as pgprot_noncached(), other parts as pgprot_writecombine() and KVM just follows them (similarly if we need a cacheable mapping). The latter has some benefits for DPDK but it's a lot more involved with having to add device-specific knowledge into the VMM. The VMM would also have to present the whole BAR contiguously to the guest even if there are different mapping attributes within the range. So a lot of MAP_FIXED uses. I'd rather leaving this decision with the guest than the VMM, it looks like more hassle to create those mappings. The VMM or the VFIO could only state write-combine and speculation allowed. -- Catalin
