During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta <pawan.kumar.gupta@xxxxxxxxxxxxxxx> --- arch/x86/kvm/vmx/vmenter.S | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 10 +++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index be275a0410a8..efa716cf4727 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> #include <asm/asm.h> +#include <asm/segment.h> #include <asm/bitsperlong.h> #include <asm/kvm_vcpu_regs.h> #include <asm/nospec-branch.h> @@ -31,6 +32,8 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif +#define GUEST_CLEAR_CPU_BUFFERS USER_CLEAR_CPU_BUFFERS + .macro VMX_DO_EVENT_IRQOFF call_insn call_target /* * Unconditionally create a stack frame, getting the correct RSP on the @@ -177,10 +180,16 @@ SYM_FUNC_START(__vmx_vcpu_run) * the 'vmx_vmexit' label below. */ .Lvmresume: + /* Mitigate CPU data sampling attacks .e.g. MDS */ + GUEST_CLEAR_CPU_BUFFERS + vmresume jmp .Lvmfail .Lvmlaunch: + /* Mitigate CPU data sampling attacks .e.g. MDS */ + GUEST_CLEAR_CPU_BUFFERS + vmlaunch jmp .Lvmfail diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c16297a49e4d..e3d0eda292c3 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7226,13 +7226,17 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still executed + * inspite of L1D Flush. This is because an extra VERW should not matter + * much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_USER_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) + /* MMIO mitigation is mutually exclusive to MDS mitigation later in asm */ mds_clear_cpu_buffers(); vmx_disable_fb_clear(vmx); -- 2.34.1
