On Fri, Apr 19, 2024 at 02:04:54PM +0200, Paolo Bonzini wrote: > On Thu, Apr 18, 2024 at 9:42 PM Michael Roth <michael.roth@xxxxxxx> wrote: > > > > This patchset is also available at: > > > > https://github.com/amdese/linux/commits/snp-host-v13 > > > > and is based on commit 4d2deb62185f (as suggested by Paolo) from: > > > > https://git.kernel.org/pub/scm/virt/kvm/kvm.git/log/?h=kvm-coco-queue > > This is pretty much ready to go into kvm-coco-queue. Let me know if > you want to do a quick v14 with the few changes I suggested, or I can > do them too. Submitted v14 based on 20cc50a0410f from latest kvm-coco-queue (bf1390326099). Hoping that way you can easily replace v13 with v14 and force-push, but let me know if you wanted to go about it a different way. > > Then the next steps are: > > 1) get the mm acks > > 2) figure out the state of patches 1-3 With latest kvm-coco-queue these patches are now in the base tree so I've dropped them from the series. > > 3) wait for more reviews of course > > 4) merge everything into kvm/next. > > Seems in good shape for a 6.10 target. Awesome! If anything needs attention just let me know. Thanks, Mike > > Paolo > > > > > Patch Layout > > ------------ > > > > 01-03: These patches are minor dependencies for this series and are already > > included in both tip/master and mainline, so are only included here > > as a stop-gap until merged from one of those trees. These are needed > > by patch #8 in this series which makes use of CC_ATTR_HOST_SEV_SNP > > > > 04: This is a small general fix-up for guest_memfd that can be applied > > independently of this series. > > > > 05-08: These patches add some basic infrastructure and introduces a new > > KVM_X86_SNP_VM vm_type to handle differences verses the existing > > KVM_X86_SEV_VM and KVM_X86_SEV_ES_VM types. > > > > 09-11: These implement the KVM API to handle the creation of a > > cryptographic launch context, encrypt/measure the initial image > > into guest memory, and finalize it before launching it. > > > > 12-17: These implement handling for various guest-generated events such > > as page state changes, onlining of additional vCPUs, etc. > > > > 18-21: These implement the gmem hooks needed to prepare gmem-allocated > > pages before mapping them into guest private memory ranges as > > well as cleaning them up prior to returning them to the host for > > use as normal memory. Because this supplants certain activities > > like issued WBINVDs during KVM MMU invalidations, there's also > > a patch to avoid duplicating that work to avoid unecessary > > overhead. > > > > 22: With all the core support in place, the patch adds a kvm_amd module > > parameter to enable SNP support. > > > > 23-26: These patches all deal with the servicing of guest requests to handle > > things like attestation, as well as some related host-management > > interfaces. > > > > > > Testing > > ------- > > > > For testing this via QEMU, use the following tree: > > > > https://github.com/amdese/qemu/commits/snp-v4-wip3 > > > > A patched OVMF is also needed due to upstream KVM no longer supporting MMIO > > ranges that are mapped as private. It is recommended you build the AmdSevX64 > > variant as it provides the kernel-hashing support present in this series: > > > > https://github.com/amdese/ovmf/commits/apic-mmio-fix1d > > > > A basic command-line invocation for SNP would be: > > > > qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2 > > -machine q35,confidential-guest-support=sev0,memory-backend=ram1 > > -object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false > > -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth= > > -bios OVMF_CODE-upstream-20240410-apic-mmio-fix1d-AmdSevX64.fd > > > > With kernel-hashing and certificate data supplied: > > > > qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2 > > -machine q35,confidential-guest-support=sev0,memory-backend=ram1 > > -object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false > > -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=,certs-path=/home/mroth/cert.blob,kernel-hashes=on > > -bios OVMF_CODE-upstream-20240410-apic-mmio-fix1d-AmdSevX64.fd > > -kernel /boot/vmlinuz-$ver > > -initrd /boot/initrd.img-$ver > > -append "root=UUID=d72a6d1c-06cf-4b79-af43-f1bac4f620f9 ro console=ttyS0,115200n8" > > > > With standard X64 OVMF package with separate image for persistent NVRAM: > > > > qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2 > > -machine q35,confidential-guest-support=sev0,memory-backend=ram1 > > -object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false > > -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth= > > -bios OVMF_CODE-upstream-20240410-apic-mmio-fix1d.fd > > -drive if=pflash,format=raw,unit=0,file=OVMF_VARS-upstream-20240410-apic-mmio-fix1d.fd,readonly=off > > > > > > Known issues / TODOs > > -------------------- > > > > * SEV-ES guests may trigger the following warning: > > > > WARNING: CPU: 151 PID: 4003 at arch/x86/kvm/mmu/mmu.c:5855 kvm_mmu_page_fault+0x33b/0x860 [kvm] > > > > It is assumed here that these will be resolved once the transition to > > PFERR_PRIVATE_ACCESS is fully completed, but if that's not the case let me > > know and will investigate further. > > > > * Base tree in some cases reports "Unpatched return thunk in use. This should > > not happen!" the first time it runs an SVM/SEV/SNP guests. This a recent > > regression upstream and unrelated to this series: > > > > https://lore.kernel.org/linux-kernel/CANpmjNOcKzEvLHoGGeL-boWDHJobwfwyVxUqMq2kWeka3N4tXA@xxxxxxxxxxxxxx/T/ > > > > * 2MB hugepage support has been dropped pending discussion on how we plan to > > re-enable it in gmem. > > > > * Host kexec should work, but there is a known issue with host kdump support > > while SNP guests are running that will be addressed as a follow-up. > > > > * SNP kselftests are currently a WIP and will be included as part of SNP > > upstreaming efforts in the near-term. > > > > > > SEV-SNP Overview > > ---------------- > > > > This part of the Secure Encrypted Paging (SEV-SNP) series focuses on the > > changes required to add KVM support for SEV-SNP. This series builds upon > > SEV-SNP guest support, which is now in mainline, and and SEV-SNP host > > initialization support, which is now in linux-next. > > > > While series provides the basic building blocks to support booting the > > SEV-SNP VMs, it does not cover all the security enhancement introduced by > > the SEV-SNP such as interrupt protection, which will added in the future. > > > > With SNP, when pages are marked as guest-owned in the RMP table, they are > > assigned to a specific guest/ASID, as well as a specific GFN with in the > > guest. Any attempts to map it in the RMP table to a different guest/ASID, > > or a different GFN within a guest/ASID, will result in an RMP nested page > > fault. > > > > Prior to accessing a guest-owned page, the guest must validate it with a > > special PVALIDATE instruction which will set a special bit in the RMP table > > for the guest. This is the only way to set the validated bit outside of the > > initial pre-encrypted guest payload/image; any attempts outside the guest to > > modify the RMP entry from that point forward will result in the validated > > bit being cleared, at which point the guest will trigger an exception if it > > attempts to access that page so it can be made aware of possible tampering. > > > > One exception to this is the initial guest payload, which is pre-validated > > by the firmware prior to launching. The guest can use Guest Message requests > > to fetch an attestation report which will include the measurement of the > > initial image so that the guest can verify it was booted with the expected > > image/environment. > > > > After boot, guests can use Page State Change requests to switch pages > > between shared/hypervisor-owned and private/guest-owned to share data for > > things like DMA, virtio buffers, and other GHCB requests. > > > > In this implementation of SEV-SNP, private guest memory is managed by a new > > kernel framework called guest_memfd (gmem). With gmem, a new > > KVM_SET_MEMORY_ATTRIBUTES KVM ioctl has been added to tell the KVM > > MMU whether a particular GFN should be backed by shared (normal) memory or > > private (gmem-allocated) memory. To tie into this, Page State Change > > requests are forward to userspace via KVM_EXIT_VMGEXIT exits, which will > > then issue the corresponding KVM_SET_MEMORY_ATTRIBUTES call to set the > > private/shared state in the KVM MMU. > > > > The gmem / KVM MMU hooks implemented in this series will then update the RMP > > table entries for the backing PFNs to set them to guest-owned/private when > > mapping private pages into the guest via KVM MMU, or use the normal KVM MMU > > handling in the case of shared pages where the corresponding RMP table > > entries are left in the default shared/hypervisor-owned state. > > > > Feedback/review is very much appreciated! > > > > -Mike > > > > > > Changes since v12: > > > > * rebased to latest kvm-coco-queue branch (commit 4d2deb62185f) > > * add more input validation for SNP_LAUNCH_START, especially for handling > > things like MBO/MBZ policy bits, and API major/minor minimums. (Paolo) > > * block SNP KVM instances from being able to run legacy SEV commands (Paolo) > > * don't attempt to measure VMSA for vcpu 0/BSP before the others, let > > userspace deal with the ordering just like with SEV-ES (Paolo) > > * fix up docs for SNP_LAUNCH_FINISH (Paolo) > > * introduce svm->sev_es.snp_has_guest_vmsa flag to better distinguish > > handling for guest-mapped vs non-guest-mapped VMSAs, rename > > 'snp_ap_create' flag to 'snp_ap_waiting_for_reset' (Paolo) > > * drop "KVM: SEV: Use a VMSA physical address variable for populating VMCB" > > as it is no longer needed due to above VMSA rework > > * replace pr_debug_ratelimited() messages for RMP #NPFs with a single trace > > event > > * handle transient PSMASH_FAIL_INUSE return codes in kvm_gmem_invalidate(), > > switch to WARN_ON*()'s to indicate remaining error cases are not expected > > and should not be seen in practice. (Paolo) > > * add a cond_resched() in kvm_gmem_invalidate() to avoid soft lock-ups when > > cleaning up large guest memory ranges. > > * rename VLEK_REQUIRED to VCEK_DISABLE. it's be more applicable if another > > key type ever gets added. > > * don't allow attestation to be paused while an attestation request is > > being processed by firmware (Tom) > > * add missing Documentation entry for SNP_VLEK_LOAD > > * collect Reviewed-by's from Paolo and Tom > > > > Changes since v11: > > > > * Rebase series on kvm-coco-queue and re-work to leverage more > > infrastructure between SNP/TDX series. > > * Drop KVM_SNP_INIT in favor of the new KVM_SEV_INIT2 interface introduced > > here (Paolo): > > https://lore.kernel.org/lkml/20240318233352.2728327-1-pbonzini@xxxxxxxxxx/ > > * Drop exposure API fields related to things like VMPL levels, migration > > agents, etc., until they are actually supported/used (Sean) > > * Rework KVM_SEV_SNP_LAUNCH_UPDATE handling to use a new > > kvm_gmem_populate() interface instead of copying data directly into > > gmem-allocated pages (Sean) > > * Add support for SNP_LOAD_VLEK, rework the SNP_SET_CONFIG_{START,END} to > > have simpler semantics that are applicable to management of SNP_LOAD_VLEK > > updates as well, rename interfaces to the now more appropriate > > SNP_{PAUSE,RESUME}_ATTESTATION > > * Fix up documentation wording and do print warnings for > > userspace-triggerable failures (Peter, Sean) > > * Fix a race with AP_CREATION wake-up events (Jacob, Sean) > > * Fix a memory leak with VMSA pages (Sean) > > * Tighten up handling of RMP page faults to better distinguish between real > > and spurious cases (Tom) > > * Various patch/documentation rewording, cleanups, etc. > > > > > > ---------------------------------------------------------------- > > Ashish Kalra (1): > > KVM: SEV: Avoid WBINVD for HVA-based MMU notifications for SNP > > > > Borislav Petkov (AMD) (3): > > [TEMP] x86/kvm/Kconfig: Have KVM_AMD_SEV select ARCH_HAS_CC_PLATFORM > > [TEMP] x86/cc: Add cc_platform_set/_clear() helpers > > [TEMP] x86/CPU/AMD: Track SNP host status with cc_platform_*() > > > > Brijesh Singh (10): > > KVM: SEV: Add GHCB handling for Hypervisor Feature Support requests > > KVM: SEV: Add KVM_SEV_SNP_LAUNCH_START command > > KVM: SEV: Add KVM_SEV_SNP_LAUNCH_UPDATE command > > KVM: SEV: Add KVM_SEV_SNP_LAUNCH_FINISH command > > KVM: SEV: Add support to handle GHCB GPA register VMGEXIT > > KVM: SEV: Add support to handle MSR based Page State Change VMGEXIT > > KVM: SEV: Add support to handle Page State Change VMGEXIT > > KVM: SEV: Add support to handle RMP nested page faults > > KVM: SVM: Add module parameter to enable SEV-SNP > > KVM: SEV: Provide support for SNP_GUEST_REQUEST NAE event > > > > Michael Roth (10): > > KVM: guest_memfd: Fix PTR_ERR() handling in __kvm_gmem_get_pfn() > > KVM: SEV: Select KVM_GENERIC_PRIVATE_MEM when CONFIG_KVM_AMD_SEV=y > > KVM: SEV: Add initial SEV-SNP support > > KVM: SEV: Add support for GHCB-based termination requests > > KVM: SEV: Implement gmem hook for initializing private pages > > KVM: SEV: Implement gmem hook for invalidating private pages > > KVM: x86: Implement gmem hook for determining max NPT mapping level > > crypto: ccp: Add the SNP_VLEK_LOAD command > > crypto: ccp: Add the SNP_{PAUSE,RESUME}_ATTESTATION commands > > KVM: SEV: Provide support for SNP_EXTENDED_GUEST_REQUEST NAE event > > > > Tom Lendacky (2): > > KVM: SEV: Add support to handle AP reset MSR protocol > > KVM: SEV: Support SEV-SNP AP Creation NAE event > > > > Documentation/virt/coco/sev-guest.rst | 69 +- > > Documentation/virt/kvm/api.rst | 73 + > > .../virt/kvm/x86/amd-memory-encryption.rst | 88 +- > > arch/x86/coco/core.c | 52 + > > arch/x86/include/asm/kvm_host.h | 2 + > > arch/x86/include/asm/sev-common.h | 22 +- > > arch/x86/include/asm/sev.h | 19 +- > > arch/x86/include/asm/svm.h | 9 +- > > arch/x86/include/uapi/asm/kvm.h | 39 + > > arch/x86/kernel/cpu/amd.c | 38 +- > > arch/x86/kernel/cpu/mtrr/generic.c | 2 +- > > arch/x86/kernel/sev.c | 10 - > > arch/x86/kvm/Kconfig | 4 + > > arch/x86/kvm/mmu.h | 2 - > > arch/x86/kvm/mmu/mmu.c | 1 + > > arch/x86/kvm/svm/sev.c | 1444 +++++++++++++++++++- > > arch/x86/kvm/svm/svm.c | 39 +- > > arch/x86/kvm/svm/svm.h | 50 + > > arch/x86/kvm/trace.h | 31 + > > arch/x86/kvm/x86.c | 19 +- > > arch/x86/virt/svm/sev.c | 106 +- > > drivers/crypto/ccp/sev-dev.c | 85 +- > > drivers/iommu/amd/init.c | 4 +- > > include/linux/cc_platform.h | 12 + > > include/linux/psp-sev.h | 4 +- > > include/uapi/linux/kvm.h | 28 + > > include/uapi/linux/psp-sev.h | 39 + > > include/uapi/linux/sev-guest.h | 9 + > > virt/kvm/guest_memfd.c | 8 +- > > 29 files changed, 2229 insertions(+), 79 deletions(-) > > > > >
