> Yes, it would be helpful to confirm what's going on.
Sean was right. It turns out that the actual cause of my example was that when doing ioremap,
the guest OS will configure the PAT based on the value of guest MTRRs.
The key function is #pat_x_mtrr_type().
In my example, the ivshmem driver tried to ioremap the PCI BAR 2 region as WB.
However, for some reason during the VM boot process, OVMF (the BIOS I was using)
set the corresponding guest MTRRs as UC (Interestingly, SeaBIOS doesn't do this).
Therefore, #pat_x_mtrr_type() determined that the actual memory type was UC.
As a result, the guest OS set the corresponding PAT as UC.
This was why ivshmem was not cacheable before removing the guest MTRR entry.
After removing the guest MTRR entry, #pat_x_mtrr_type() would return WB.
So this was why ivshmem became cacheable after removing the guest MTRR entry.
> What test(s) did you run to determine whether or not the memory was truly cacheable?
> KVM emulates the MTRR MSRs themselves, e.g. the guest can read and write MTRRs,
> and the guest will _think_ memory has a certain memtype, but that doesn't necessarily
> have any impact on the memtype used by the CPU.
Thanks for the clarification. I used a memcpy benchmark (size 500M) to determine whether or not the memory was cacheable.
When the memory was not cacheable, the benchmark took several seconds to finish.
When the memory was cacheable, the benchmark took several milliseconds to finish.
> Heh, this isn't opinion. Unless you're running a very specific 10-year old kernel,
> or a custom KVM build, KVM simply out doesn't propagate guest MTRRs into NPT.
>
> And unless your setup also has non-coherent DMA attached to the device, KVM doesn't
> honor guest MTRRs for EPT either (AFAICT, QEMU ivshmem doesn't require VFIO).
>
> It's definitely possible that disabling a guest MTRR resulted in memory becoming
> cacheable, but unless there's some very, very magical code hiding, it's not because
> KVM actually fully virtualizes guest MTRRs on AMD.
>
> E.g. before commit 9a3768191d95 ("KVM: x86/mmu: Zap SPTEs on MTRR update iff guest
> MTRRs are honored"), which hasn't even made its way to Linus (or Paolo's) tree yet,
> KVM unnecessarily zapped all NPT entries on MTRR changes. Zapping NPT entries
> could have cleared some weird TLB state, or perhaps even wiped out buggy KVM NPT
> entries.
>
> And on AMD, hardware virtualizes gCR0.CD, i.e. puts the caches into no-fill mode
> when guest CR0.CD=1. But Intel CPUs completely ignore guest CR0.CD, i.e. punt it
> to software, and under QEMU, for all intents and purposes KVM never honors guest
> CR0.CD for VMX. It's seems highly quite unlikely that something in the guest left
> CR0.CD=1, but it's possible. And then the guest kernel's process of toggling
> CR0.CD when doing MTRR updates would end up clearing CR0.CD and thus re-enable
> caching.
>
>> The thing was that I could not find any KVM code related to emulating guest
>> MTRRs on AMD platforms, which was the reason why I decided to send the
>> initial email asking about it.
>>
>> I found this in the AMD64 Architecture Programmer’s Manual Volumes 1–5 (page
>> 553):
>>
>> "Table 15-19 shows how guest and host PAT types are combined into an
>> effective PAT type. When interpreting this table, recall (a) that guest and
>> host PAT types are not combined when nested paging is disabled and (b) that
>> the intent is for the VMM to use its PAT type to simulate guest MTRRs.”
>>
>> Does this mean that AMD expects the VMM to emulate the effect of guest MTRRs
>> by altering the host PAT types?
>
> Yes. Which is exactly what KVM did in commit 3c2e7f7de324 ("KVM: SVM: use NPT
> page attributes"), which was a reverted a few months after it was introduced.
Again, thanks for the clarification!
