Overhaul TDP MMU's handling of zapping and TLB flushing to reduce the
number of TLB flushes, and to clean up the zapping code. The final patch
realizes the biggest change, which is to use RCU to defer any TLB flush
due to zapping a SP to the caller. The largest cleanup is to separate the
flows for zapping roots (zap _everything_), zapping leaf SPTEs (zap guest
mappings for whatever reason), and zapping a specific SP (NX recovery).
They're currently smushed into a single zap_gfn_range(), which was a good
idea at the time, but became a mess when trying to handle the different
rules, e.g. TLB flushes aren't needed when zapping a root because KVM can
safely zap a root if and only if it's unreachable.
For booting an 8 vCPU, remote_tlb_flush (requests) goes from roughly
180 (600) to 130 (215).
Please don't apply patches 02 and 03, they've been posted elsehwere and by
other people. I included them here because some of the patches have
pseudo-dependencies on their changes. Patch 01 is also posted separately.
I had a brain fart and sent it out realizing that doing so would lead to
oddities.
Hou Wenlong (1):
KVM: x86/mmu: Skip tlb flush if it has been done in zap_gfn_range()
Sean Christopherson (27):
KVM: x86/mmu: Use yield-safe TDP MMU root iter in MMU notifier
unmapping
KVM: x86/mmu: Remove spurious TLB flushes in TDP MMU zap collapsible
path
KVM: x86/mmu: Retry page fault if root is invalidated by memslot
update
KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP
MMU
KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush
logic
KVM: x86/mmu: Document that zapping invalidated roots doesn't need to
flush
KVM: x86/mmu: Drop unused @kvm param from kvm_tdp_mmu_get_root()
KVM: x86/mmu: Require mmu_lock be held for write in unyielding root
iter
KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU
root
KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP
removal
KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier
change_spte
KVM: x86/mmu: Drop RCU after processing each root in MMU notifier
hooks
KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCU
KVM: x86/mmu: Take TDP MMU roots off list when invalidating all roots
KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic path
KVM: x86/mmu: Terminate yield-friendly walk if invalid root observed
KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw
vals
KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recovery
KVM: x86/mmu: Use common TDP MMU zap helper for MMU notifier unmap
hook
KVM: x86/mmu: Add TDP MMU helper to zap a root
KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMU
KVM: x86/mmu: Use "zap root" path for "slow" zap of all TDP MMU SPTEs
KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow page
KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU range
KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()
KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU
resched
KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow
pages
arch/x86/kvm/mmu/mmu.c | 74 +++--
arch/x86/kvm/mmu/mmu_internal.h | 7 +-
arch/x86/kvm/mmu/paging_tmpl.h | 3 +-
arch/x86/kvm/mmu/tdp_iter.c | 6 +-
arch/x86/kvm/mmu/tdp_iter.h | 15 +-
arch/x86/kvm/mmu/tdp_mmu.c | 526 +++++++++++++++++++-------------
arch/x86/kvm/mmu/tdp_mmu.h | 48 +--
7 files changed, 406 insertions(+), 273 deletions(-)
--
2.34.0.rc2.393.gf8c9666880-goog
