KVM x86/mmu changes for 6.4. The highlights are optimizations from Lai (.invlpg(), .sync_page(), and rmaps) and Vipin (A/D harvesting). I am also planning on sending the "persistent TDP MMU roots" patch[*] for 6.4-rc1, just waiting a few more days to give syzbot extra time to beat on v2, and to settle on whether to guard VM desctruction with mmu_lock or RCU. [*] https://lore.kernel.org/all/20230421214946.2571580-1-seanjc@xxxxxxxxxx The following changes since commit d8708b80fa0e6e21bc0c9e7276ad0bccef73b6e7: KVM: Change return type of kvm_arch_vm_ioctl() to "int" (2023-03-16 10:18:07 -0400) are available in the Git repository at: https://github.com/kvm-x86/linux.git tags/kvm-x86-mmu-6.4 for you to fetch changes up to 9ed3bf411226f446a9795f2b49a15b9df98d7cf5: KVM: x86/mmu: Move filling of Hyper-V's TLB range struct into Hyper-V code (2023-04-10 15:17:29 -0700) ---------------------------------------------------------------- KVM x86 MMU changes for 6.4: - Tweak FNAME(sync_spte) to avoid unnecessary writes+flushes when the guest is only adding new PTEs - Overhaul .sync_page() and .invlpg() to share the .sync_page() implementation, i.e. utilize .sync_page()'s optimizations when emulating invalidations - Clean up the range-based flushing APIs - Revamp the TDP MMU's reaping of Accessed/Dirty bits to clear a single A/D bit using a LOCK AND instead of XCHG, and skip all of the "handle changed SPTE" overhead associated with writing the entire entry - Track the number of "tail" entries in a pte_list_desc to avoid having to walk (potentially) all descriptors during insertion and deletion, which gets quite expensive if the guest is spamming fork() - Misc cleanups ---------------------------------------------------------------- David Matlack (3): KVM: x86/mmu: Collapse kvm_flush_remote_tlbs_with_{range,address}() together KVM: x86/mmu: Rename kvm_flush_remote_tlbs_with_address() KVM: x86/mmu: Use gfn_t in kvm_flush_remote_tlbs_range() Lai Jiangshan (14): KVM: x86/mmu: Use 64-bit address to invalidate to fix a subtle bug KVM: x86/mmu: Move the check in FNAME(sync_page) as kvm_sync_page_check() KVM: x86/mmu: Check mmu->sync_page pointer in kvm_sync_page_check() KVM: x86/mmu: Set mmu->sync_page as NULL for direct paging KVM: x86/mmu: Move the code out of FNAME(sync_page)'s loop body into mmu.c KVM: x86/mmu: Reduce the update to the spte in FNAME(sync_spte) kvm: x86/mmu: Use KVM_MMU_ROOT_XXX for kvm_mmu_invalidate_addr() KVM: x86/mmu: Use kvm_mmu_invalidate_addr() in kvm_mmu_invpcid_gva() KVM: x86/mmu: Use kvm_mmu_invalidate_addr() in nested_ept_invalidate_addr() KVM: x86/mmu: Allow the roots to be invalid in FNAME(invlpg) KVM: x86/mmu: Remove FNAME(invlpg) and use FNAME(sync_spte) to update vTLB instead. kvm: x86/mmu: Remove @no_dirty_log from FNAME(prefetch_gpte) KVM: x86/mmu: Skip calling mmu->sync_spte() when the spte is 0 KVM: x86/mmu: Track tail count in pte_list_desc to optimize guest fork() Mathias Krause (1): KVM: x86/mmu: Fix comment typo Paolo Bonzini (1): KVM: x86/mmu: Avoid indirect call for get_cr3 Sean Christopherson (6): KVM: x86/mmu: Sanity check input to kvm_mmu_free_roots() KVM: x86/mmu: Rename slot rmap walkers to add clarity and clean up code KVM: x86/mmu: Replace comment with an actual lockdep assertion on mmu_lock KVM: x86/mmu: Clean up mmu.c functions that put return type on separate line KVM: x86: Rename Hyper-V remote TLB hooks to match established scheme KVM: x86/mmu: Move filling of Hyper-V's TLB range struct into Hyper-V code Vipin Sharma (13): KVM: x86/mmu: Add a helper function to check if an SPTE needs atomic write KVM: x86/mmu: Use kvm_ad_enabled() to determine if TDP MMU SPTEs need wrprot KVM: x86/mmu: Consolidate Dirty vs. Writable clearing logic in TDP MMU KVM: x86/mmu: Atomically clear SPTE dirty state in the clear-dirty-log flow KVM: x86/mmu: Drop access tracking checks when clearing TDP MMU dirty bits KVM: x86/mmu: Bypass __handle_changed_spte() when clearing TDP MMU dirty bits KVM: x86/mmu: Remove "record_dirty_log" in __tdp_mmu_set_spte() KVM: x86/mmu: Clear only A-bit (if enabled) when aging TDP MMU SPTEs KVM: x86/mmu: Drop unnecessary dirty log checks when aging TDP MMU SPTEs KVM: x86/mmu: Bypass __handle_changed_spte() when aging TDP MMU SPTEs KVM: x86/mmu: Remove "record_acc_track" in __tdp_mmu_set_spte() KVM: x86/mmu: Remove handle_changed_spte_dirty_log() KVM: x86/mmu: Merge all handle_changed_pte*() functions arch/x86/include/asm/kvm-x86-ops.h | 4 +- arch/x86/include/asm/kvm_host.h | 32 +-- arch/x86/kvm/kvm_onhyperv.c | 33 ++- arch/x86/kvm/kvm_onhyperv.h | 5 +- arch/x86/kvm/mmu/mmu.c | 506 ++++++++++++++++++++++--------------- arch/x86/kvm/mmu/mmu_internal.h | 8 +- arch/x86/kvm/mmu/paging_tmpl.h | 224 +++++----------- arch/x86/kvm/mmu/spte.c | 2 +- arch/x86/kvm/mmu/tdp_iter.h | 48 +++- arch/x86/kvm/mmu/tdp_mmu.c | 215 ++++++---------- arch/x86/kvm/svm/svm_onhyperv.h | 5 +- arch/x86/kvm/vmx/nested.c | 5 +- arch/x86/kvm/vmx/vmx.c | 5 +- arch/x86/kvm/x86.c | 4 +- 14 files changed, 522 insertions(+), 574 deletions(-)
