On Wed, Dec 11, 2019 at 3:08 AM Marc Zyngier <maz@xxxxxxxxxx> wrote: > > Hi Gurchetan, > > I don't know anything about graphics API, so please bear with me. Vulkan exposes the concept of memory types to userspace. These memory types describe the memory properties of a heap. For example, any memory type with the HOST_COHERENT_BIT specifies the application isn't required to send certain cache management commands (vkFlushMappedMemoryRanges, vkInvalidateMappedMemoryRanges). 1) HOST_CACHED => cached, no cache management 2) HOST_COHERENT_BIT => write combine, no cache management, no snooping 3) HOST_COHERENT_BIT | HOST_CACHED => cached, gpu snoops cpu caches, no cache management Here's some more reading on that: https://static.docs.arm.com/100019/0100/arm_mali_application_developer_best_practices_developer_guide_100019_0100_00_en2.pdf (1) and (3) aren't too difficult -- just use KVM_SET_USER_MEMORY_REGION. (2) is, since it may lead to inconsistent mappings. vmware has a very nice emulated solution for this: https://lwn.net/Articles/804114/ We're planning on building off vmware's solution, and if possible, add some additional optimizations. > > On 2019-12-11 01:32, Gurchetan Singh wrote: > > Hi, > > > > We're trying to implement Vulkan with virtio-gpu, and that API > > exposes > > the difference between cached and uncached mappings to userspace > > (i.e, > > some older GPUs can't snoop the CPU's caches). > > > > We need to make sure that the guest and host caching attributes are > > aligned, or there's a proper API between the virtio driver and device > > to ensure coherence. > > I think you trying to cross two barriers at once here. > > Virtio is always coherent between host and guest, and the guest should > use cacheable mappings. That's at least my expectation, and I don't > know of any exception to this rule. Where is this ruled stated? We may need to modify the spec. At the very least, we need anything mapped on the host with HOST_COHERENT_BIT only to be write combine in the guest. However, there's an additional caveat: on x86, cache management (clflush) is available to the guest. So it's possible to map host cached memory as write-combine, and import guest memory to Vulkan. This is an optimization limited to non-zero amount of x86 devices. So, something like arch_does_guest_attribute_matter() or arch_can_guest_flush() in the guest kernel would be useful. But it doesn't sound like this is readily available? > You have then the coherency of the physical device, and that's a host > kernel (and maybe userspace) matter. Why should the guest ever know > about > this constraint? > > > One issue that needs to be addressed is the caching policy is > > variable > > dependent on the VM configuration and architecture. For example, on > > x86, it looks like a MTRR controls whether the guest caching > > attribute > > predominates[1]. On ARM, it looks like the MMU registers control > > whether the guest can override the host attribute, but in general > > it's > > most restrictive attribute that makes a difference[2]. Let me if > > that's incorrect. > > For ARM, this is true up to ARMv8.3. Starting with ARMv8.4, FWB gives > the hypervisor the opportunity to force memory mappings as cacheable > write-back. None of that is visible to userspace, thankfully. Is there some open-source code available on ARM showing how the MMU is configured (which forces guest attribute override)? > > > I'm wondering if there's some standard kernel API to query such > > attributes. For example, something like > > arch_does_guest_attribute_matter() or arch_can_guest_flush() would do > > the trick. Without this, we may need to introduce VIRTIO_GPU_F_* > > flags set by the host, but that may make the already giant QEMU > > command line even bigger. > > If something has to manage the coherency, it should be the host that > knows how the memory traffic flows between host and guest, and apply > cache management as required. Note that on arm64, cache management > instructions are available from userspace. On systems that are > fully coherent, they are expected to be little more than NOPs. > > From the above, it is pretty obvious that I don't understand what > problem you are trying to solve. Maybe you could explain how > you envisage things to work, who maps what where, and the expected > semantics. Once we have a common understanding of the problem, > maybe we can think of a decent solution. > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny...
