On 3/3/22 20:38, Paolo Bonzini wrote:
Overhaul TDP MMU's handling of zapping and TLB flushing to reduce the
number of TLB flushes, fix soft lockups and RCU stalls, avoid blocking
vCPUs for long durations while zapping paging structure, and to clean up
the zapping code.
The largest cleanup is to separate the flows for zapping roots (zap
_everything_), zapping leaf SPTEs (zap guest mappings for whatever reason),
and zapping a specific SP (NX recovery). They're currently smushed into a
single zap_gfn_range(), which was a good idea at the time, but became a
mess when trying to handle the different rules, e.g. TLB flushes aren't
needed when zapping a root because KVM can safely zap a root if and only
if it's unreachable.
To solve the soft lockups, stalls, and vCPU performance issues:
- Defer remote TLB flushes to the caller when zapping TDP MMU shadow
pages by relying on RCU to ensure the paging structure isn't freed
until all vCPUs have exited the guest.
- Allowing yielding when zapping TDP MMU roots in response to the root's
last reference being put. This requires a bit of trickery to ensure
the root is reachable via mmu_notifier, but it's not too gross.
- Zap roots in two passes to avoid holding RCU for potential hundreds of
seconds when zapping guest with terabytes of memory that is backed
entirely by 4kb SPTEs.
- Zap defunct roots asynchronously via the common work_queue so that a
vCPU doesn't get stuck doing the work if the vCPU happens to drop the
last reference to a root.
The selftest at the end allows populating a guest with the max amount of
memory allowed by the underlying architecture. The most I've tested is
~64tb (MAXPHYADDR=46) as I don't have easy access to a system with
MAXPHYADDR=52. The selftest compiles on arm64 and s390x, but otherwise
hasn't been tested outside of x86-64. It will hopefully do something
useful as is, but there's a non-zero chance it won't get past init with
a high max memory. Running on x86 without the TDP MMU is comically slow.
Testing: passes kvm-unit-tests and guest installation tests on Intel.
Haven't yet run AMD or selftests.
Thanks,
Paolo
v4:
- collected reviews and typo fixes (plus some typo fixes of my own)
- new patches to simplify reader invariants: they are not allowed to
acquire references to invalid roots
- new version of "Allow yielding when zapping GFNs for defunct TDP MMU
root", simplifying the atomic a bit by 1) using xchg and relying on
its implicit memory barriers 2) relying on readers to have the same
behavior for the three stats refcount=0/valid, refcount=0/invalid,
refcount=1/invalid (see previous point)
- switch zapping of invalidated roots to asynchronous workers on a
per-VM workqueue, fixing a bug in v3 where the extra reference added
by kvm_tdp_mmu_put_root could be given back twice. This also replaces
"KVM: x86/mmu: Use common iterator for walking invalid TDP MMU roots"
in v3, since it gets rid of next_invalidated_root() in a different way.
- because of the previous point, most of the logic in v3's "KVM: x86/mmu:
Zap defunct roots via asynchronous worker" moves to the earlier patch
"KVM: x86/mmu: Zap invalidated roots via asynchronous worker"
v3:
- Drop patches that were applied.
- Rebase to latest kvm/queue.
- Collect a review. [David]
- Use helper instead of goto to zap roots in two passes. [David]
- Add patches to disallow REMOVED "old" SPTE when atomically
setting SPTE.
Paolo Bonzini (5):
KVM: x86/mmu: only perform eager page splitting on valid roots
KVM: x86/mmu: do not allow readers to acquire references to invalid roots
KVM: x86/mmu: Zap invalidated roots via asynchronous worker
KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU root
KVM: x86/mmu: Zap defunct roots via asynchronous worker
Sean Christopherson (25):
KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP MMU
KVM: x86/mmu: Fix wrong/misleading comments in TDP MMU fast zap
KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush logic
KVM: x86/mmu: Document that zapping invalidated roots doesn't need to flush
KVM: x86/mmu: Require mmu_lock be held for write in unyielding root iter
KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP removal
KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier change_spte
KVM: x86/mmu: Drop RCU after processing each root in MMU notifier hooks
KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCU
KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic path
KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw values
KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recovery
KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMU
KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow page
KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU range
KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()
KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU resched
KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pages
KVM: x86/mmu: Zap roots in two passes to avoid inducing RCU stalls
KVM: x86/mmu: Check for a REMOVED leaf SPTE before making the SPTE
KVM: x86/mmu: WARN on any attempt to atomically update REMOVED SPTE
KVM: selftests: Move raw KVM_SET_USER_MEMORY_REGION helper to utils
KVM: selftests: Split out helper to allocate guest mem via memfd
KVM: selftests: Define cpu_relax() helpers for s390 and x86
KVM: selftests: Add test to populate a VM with the max possible guest mem
arch/x86/include/asm/kvm_host.h | 2 +
arch/x86/kvm/mmu/mmu.c | 49 +-
arch/x86/kvm/mmu/mmu_internal.h | 15 +-
arch/x86/kvm/mmu/tdp_iter.c | 6 +-
arch/x86/kvm/mmu/tdp_iter.h | 15 +-
arch/x86/kvm/mmu/tdp_mmu.c | 559 +++++++++++-------
arch/x86/kvm/mmu/tdp_mmu.h | 26 +-
tools/testing/selftests/kvm/.gitignore | 1 +
tools/testing/selftests/kvm/Makefile | 3 +
.../selftests/kvm/include/kvm_util_base.h | 5 +
.../selftests/kvm/include/s390x/processor.h | 8 +
.../selftests/kvm/include/x86_64/processor.h | 5 +
tools/testing/selftests/kvm/lib/kvm_util.c | 66 ++-
.../selftests/kvm/max_guest_memory_test.c | 292 +++++++++
.../selftests/kvm/set_memory_region_test.c | 35 +-
15 files changed, 794 insertions(+), 293 deletions(-)
create mode 100644 tools/testing/selftests/kvm/max_guest_memory_test.c
Queued, thanks.
Paolo
