On Wed, 11 Dec 2019 23:34:30 +0000, Gurchetan Singh <gurchetansingh@xxxxxxxxxxxx> wrote: > > On Wed, Dec 11, 2019 at 3:08 AM Marc Zyngier <maz@xxxxxxxxxx> wrote: > > > > Hi Gurchetan, > > > > I don't know anything about graphics API, so please bear with me. > > Vulkan exposes the concept of memory types to userspace. These memory Key word: userspace. > types describe the memory properties of a heap. For example, any > memory type with the HOST_COHERENT_BIT specifies the application isn't > required to send certain cache management commands > (vkFlushMappedMemoryRanges, vkInvalidateMappedMemoryRanges). > > 1) HOST_CACHED => cached, no cache management > 2) HOST_COHERENT_BIT => write combine, no cache management, no snooping > 3) HOST_COHERENT_BIT | HOST_CACHED => cached, gpu snoops cpu caches, > no cache management And that's absolutely fine between the CPU and the GPU. I don't see why the guest, which is supposed to use a PV (virtio) driver should be impacted by any of this. > Here's some more reading on that: > > https://static.docs.arm.com/100019/0100/arm_mali_application_developer_best_practices_developer_guide_100019_0100_00_en2.pdf Sorry, but seeing the words "Mali" and "best practices" in the same sentence is just hilarious. I haven't read any further. > (1) and (3) aren't too difficult -- just use > KVM_SET_USER_MEMORY_REGION. (2) is, since it may lead to > inconsistent mappings. How is that relevant to the above? Most importantly, why should the guest be in *any way* influenced by the coherency (or lack thereof) of the GPU driver on the host? The guest should see a cache-coherent PV driver, and the host does its best to cope with it. After all, that host driver knows exactly what level of coherency it has to deal with. > vmware has a very nice emulated solution for this: > > https://lwn.net/Articles/804114/ > > We're planning on building off vmware's solution, and if possible, add > some additional optimizations. Well, let's try to walk before we run, shall we? > > On 2019-12-11 01:32, Gurchetan Singh wrote: > > > Hi, > > > > > > We're trying to implement Vulkan with virtio-gpu, and that API > > > exposes > > > the difference between cached and uncached mappings to userspace > > > (i.e, > > > some older GPUs can't snoop the CPU's caches). > > > > > > We need to make sure that the guest and host caching attributes are > > > aligned, or there's a proper API between the virtio driver and device > > > to ensure coherence. > > > > I think you trying to cross two barriers at once here. > > > > Virtio is always coherent between host and guest, and the guest should > > use cacheable mappings. That's at least my expectation, and I don't > > know of any exception to this rule. > > Where is this ruled stated? We may need to modify the spec. Sure. The spec is public, you just need to be convincing enough. > At the very least, we need anything mapped on the host with > HOST_COHERENT_BIT only to be write combine in the guest. You don't get to choose. The guest does. And you haven't explained why the guest should be involved in any way. > However, there's an additional caveat: on x86, cache management > (clflush) is available to the guest. So it's possible to map host > cached memory as write-combine, and import guest memory to Vulkan. > This is an optimization limited to non-zero amount of x86 devices. If you say so. I have no idea. > So, something like arch_does_guest_attribute_matter() or > arch_can_guest_flush() in the guest kernel would be useful. But it > doesn't sound like this is readily available? Given that you haven't given the definition of what this is supposed to do, and that you keep giving solutions without having described the problem, I have no answer to give you. > > You have then the coherency of the physical device, and that's a host > > kernel (and maybe userspace) matter. Why should the guest ever know > > about > > this constraint? > > > > > One issue that needs to be addressed is the caching policy is > > > variable > > > dependent on the VM configuration and architecture. For example, on > > > x86, it looks like a MTRR controls whether the guest caching > > > attribute > > > predominates[1]. On ARM, it looks like the MMU registers control > > > whether the guest can override the host attribute, but in general > > > it's > > > most restrictive attribute that makes a difference[2]. Let me if > > > that's incorrect. > > > > For ARM, this is true up to ARMv8.3. Starting with ARMv8.4, FWB gives > > the hypervisor the opportunity to force memory mappings as cacheable > > write-back. None of that is visible to userspace, thankfully. > > Is there some open-source code available on ARM showing how the MMU is > configured (which forces guest attribute override)? The kernel is all yours, and the ARMv8 ARM is an entertaining read. > > > I'm wondering if there's some standard kernel API to query such > > > attributes. For example, something like > > > arch_does_guest_attribute_matter() or arch_can_guest_flush() would do > > > the trick. Without this, we may need to introduce VIRTIO_GPU_F_* > > > flags set by the host, but that may make the already giant QEMU > > > command line even bigger. > > > > If something has to manage the coherency, it should be the host that > > knows how the memory traffic flows between host and guest, and apply > > cache management as required. Note that on arm64, cache management > > instructions are available from userspace. On systems that are > > fully coherent, they are expected to be little more than NOPs. > > > > From the above, it is pretty obvious that I don't understand what > > problem you are trying to solve. Maybe you could explain how > > you envisage things to work, who maps what where, and the expected > > semantics. Once we have a common understanding of the problem, > > maybe we can think of a decent solution. I'm still expecting answers to the above, as this is really key. M. -- Jazz is not dead, it just smells funny.
