[AMD Public Use]
Hello Steve,
I don't think we have ever discussed supporting this command, maybe we can support it in a future follow up patch.
Thanks,
Ashish
-----Original Message-----
From: Steve Rutherford <srutherford@xxxxxxxxxx>
Sent: Thursday, January 14, 2021 6:32 PM
To: Kalra, Ashish <Ashish.Kalra@xxxxxxx>
Cc: Paolo Bonzini <pbonzini@xxxxxxxxxx>; Thomas Gleixner <tglx@xxxxxxxxxxxxx>; Ingo Molnar <mingo@xxxxxxxxxx>; H. Peter Anvin <hpa@xxxxxxxxx>; Radim Krčmář <rkrcmar@xxxxxxxxxx>; Joerg Roedel <joro@xxxxxxxxxx>; Borislav Petkov <bp@xxxxxxx>; Lendacky, Thomas <Thomas.Lendacky@xxxxxxx>; X86 ML <x86@xxxxxxxxxx>; KVM list <kvm@xxxxxxxxxxxxxxx>; LKML <linux-kernel@xxxxxxxxxxxxxxx>; Venu Busireddy <venu.busireddy@xxxxxxxxxx>; Singh, Brijesh <brijesh.singh@xxxxxxx>
Subject: Re: [PATCH v9 00/18] Add AMD SEV guest live migration support
Forgot to ask this: is there an intention to support SEND_CANCEL in a follow up patch?
On Tue, Dec 8, 2020 at 2:03 PM Ashish Kalra <Ashish.Kalra@xxxxxxx> wrote:
>
> From: Ashish Kalra <ashish.kalra@xxxxxxx>
>
> The series add support for AMD SEV guest live migration commands. To
> protect the confidentiality of an SEV protected guest memory while in
> transit we need to use the SEV commands defined in SEV API spec [1].
>
> SEV guest VMs have the concept of private and shared memory. Private
> memory is encrypted with the guest-specific key, while shared memory
> may be encrypted with hypervisor key. The commands provided by the SEV
> FW are meant to be used for the private memory only. The patch series introduces a new hypercall.
> The guest OS can use this hypercall to notify the page encryption status.
> If the page is encrypted with guest specific-key then we use SEV
> command during the migration. If page is not encrypted then fallback to default.
>
> The patch adds new ioctls KVM_{SET,GET}_PAGE_ENC_BITMAP. The ioctl can
> be used by the qemu to get the page encrypted bitmap. Qemu can consult
> this bitmap during the migration to know whether the page is encrypted.
>
> This section descibes how the SEV live migration feature is negotiated
> between the host and guest, the host indicates this feature support
> via KVM_FEATURE_CPUID. The guest firmware (OVMF) detects this feature
> and sets a UEFI enviroment variable indicating OVMF support for live
> migration, the guest kernel also detects the host support for this
> feature via cpuid and in case of an EFI boot verifies if OVMF also
> supports this feature by getting the UEFI enviroment variable and if
> it set then enables live migration feature on host by writing to a
> custom MSR, if not booted under EFI, then it simply enables the
> feature by again writing to the custom MSR. The host returns error as
> part of SET_PAGE_ENC_BITMAP ioctl if guest has not enabled live migration.
>
> A branch containing these patches is available here:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2FAMDESE%2Flinux%2Ftree%2Fsev-migration-v9&data=04%7C01%7CA
> shish.Kalra%40amd.com%7C940bfdcae0f640321ff208d8b8ed0e93%7C3dd8961fe48
> 84e608e11a82d994e183d%7C0%7C0%7C637462676023260162%7CUnknown%7CTWFpbGZ
> sb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> D%7C1000&sdata=yyEegH2gWukbKMM%2FQ%2FgkMpHacwxu7KJ0E3Q3wfxLZ%2B0%3
> D&reserved=0
>
> [1]
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeve
> loper.amd.com%2Fwp-content%2Fresources%2F55766.PDF&data=04%7C01%7C
> Ashish.Kalra%40amd.com%7C940bfdcae0f640321ff208d8b8ed0e93%7C3dd8961fe4
> 884e608e11a82d994e183d%7C0%7C0%7C637462676023270160%7CUnknown%7CTWFpbG
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&sdata=RhUoTrbJelVdD4ijE4XpZNIr%2BIMokCY6FTtlAWhcLEs%3D&a
> mp;reserved=0
>
> Changes since v8:
> - Rebasing to kvm next branch.
> - Fixed and added comments as per review feedback on v8 patches.
> - Removed implicitly enabling live migration for incoming VMs in
> in KVM_SET_PAGE_ENC_BITMAP, it is now done via KVM_SET_MSR ioctl.
> - Adds support for bypassing unencrypted guest memory regions for
> DBG_DECRYPT API calls, guest memory region encryption status in
> sev_dbg_decrypt() is referenced using the page encryption bitmap.
>
> Changes since v7:
> - Removed the hypervisor specific hypercall/paravirt callback for
> SEV live migration and moved back to calling kvm_sev_hypercall3
> directly.
> - Fix build errors as
> Reported-by: kbuild test robot <lkp@xxxxxxxxx>, specifically fixed
> build error when CONFIG_HYPERVISOR_GUEST=y and
> CONFIG_AMD_MEM_ENCRYPT=n.
> - Implicitly enabled live migration for incoming VM(s) to handle
> A->B->C->... VM migrations.
> - Fixed Documentation as per comments on v6 patches.
> - Fixed error return path in sev_send_update_data() as per comments
> on v6 patches.
>
> Changes since v6:
> - Rebasing to mainline and refactoring to the new split SVM
> infrastructre.
> - Move to static allocation of the unified Page Encryption bitmap
> instead of the dynamic resizing of the bitmap, the static allocation
> is done implicitly by extending kvm_arch_commit_memory_region() callack
> to add svm specific x86_ops which can read the userspace provided memory
> region/memslots and calculate the amount of guest RAM managed by the KVM
> and grow the bitmap.
> - Fixed KVM_SET_PAGE_ENC_BITMAP ioctl to set the whole bitmap instead
> of simply clearing specific bits.
> - Removed KVM_PAGE_ENC_BITMAP_RESET ioctl, which is now performed using
> KVM_SET_PAGE_ENC_BITMAP.
> - Extended guest support for enabling Live Migration feature by adding a
> check for UEFI environment variable indicating OVMF support for Live
> Migration feature and additionally checking for KVM capability for the
> same feature. If not booted under EFI, then we simply check for KVM
> capability.
> - Add hypervisor specific hypercall for SEV live migration by adding
> a new paravirt callback as part of x86_hyper_runtime.
> (x86 hypervisor specific runtime callbacks)
> - Moving MSR handling for MSR_KVM_SEV_LIVE_MIG_EN into svm/sev code
> and adding check for SEV live migration enabled by guest in the
> KVM_GET_PAGE_ENC_BITMAP ioctl.
> - Instead of the complete __bss_decrypted section, only specific variables
> such as hv_clock_boot and wall_clock are marked as decrypted in the
> page encryption bitmap
>
> Changes since v5:
> - Fix build errors as
> Reported-by: kbuild test robot <lkp@xxxxxxxxx>
>
> Changes since v4:
> - Host support has been added to extend KVM capabilities/feature bits to
> include a new KVM_FEATURE_SEV_LIVE_MIGRATION, which the guest can
> query for host-side support for SEV live migration and a new custom MSR
> MSR_KVM_SEV_LIVE_MIG_EN is added for guest to enable the SEV live
> migration feature.
> - Ensure that _bss_decrypted section is marked as decrypted in the
> page encryption bitmap.
> - Fixing KVM_GET_PAGE_ENC_BITMAP ioctl to return the correct bitmap
> as per the number of pages being requested by the user. Ensure that
> we only copy bmap->num_pages bytes in the userspace buffer, if
> bmap->num_pages is not byte aligned we read the trailing bits
> from the userspace and copy those bits as is. This fixes guest
> page(s) corruption issues observed after migration completion.
> - Add kexec support for SEV Live Migration to reset the host's
> page encryption bitmap related to kernel specific page encryption
> status settings before we load a new kernel by kexec. We cannot
> reset the complete page encryption bitmap here as we need to
> retain the UEFI/OVMF firmware specific settings.
>
> Changes since v3:
> - Rebasing to mainline and testing.
> - Adding a new KVM_PAGE_ENC_BITMAP_RESET ioctl, which resets the
> page encryption bitmap on a guest reboot event.
> - Adding a more reliable sanity check for GPA range being passed to
> the hypercall to ensure that guest MMIO ranges are also marked
> in the page encryption bitmap.
>
> Changes since v2:
> - reset the page encryption bitmap on vcpu reboot
>
> Changes since v1:
> - Add support to share the page encryption between the source and target
> machine.
> - Fix review feedbacks from Tom Lendacky.
> - Add check to limit the session blob length.
> - Update KVM_GET_PAGE_ENC_BITMAP icotl to use the base_gfn instead of
> the memory slot when querying the bitmap.
>
> Ashish Kalra (7):
> KVM: SVM: Add support for static allocation of unified Page Encryption
> Bitmap.
> KVM: x86: Introduce new KVM_FEATURE_SEV_LIVE_MIGRATION feature &
> Custom MSR.
> EFI: Introduce the new AMD Memory Encryption GUID.
> KVM: x86: Add guest support for detecting and enabling SEV Live
> Migration feature.
> KVM: x86: Mark _bss_decrypted section variables as decrypted in page
> encryption bitmap.
> KVM: x86: Add kexec support for SEV Live Migration.
> KVM: SVM: Enable SEV live migration feature implicitly on Incoming
> VM(s).
>
> Brijesh Singh (11):
> KVM: SVM: Add KVM_SEV SEND_START command
> KVM: SVM: Add KVM_SEND_UPDATE_DATA command
> KVM: SVM: Add KVM_SEV_SEND_FINISH command
> KVM: SVM: Add support for KVM_SEV_RECEIVE_START command
> KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command
> KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command
> KVM: x86: Add AMD SEV specific Hypercall3
> KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall
> KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl
> mm: x86: Invoke hypercall when page encryption status is changed
> KVM: x86: Introduce KVM_SET_PAGE_ENC_BITMAP ioctl
>
> Ashish Kalra (7):
> KVM: SVM: Add support for static allocation of unified Page Encryption
> Bitmap.
> KVM: x86: Introduce new KVM_FEATURE_SEV_LIVE_MIGRATION feature &
> Custom MSR.
> EFI: Introduce the new AMD Memory Encryption GUID.
> KVM: x86: Add guest support for detecting and enabling SEV Live
> Migration feature.
> KVM: x86: Mark _bss_decrypted section variables as decrypted in page
> encryption bitmap.
> KVM: x86: Add kexec support for SEV Live Migration.
> KVM: SVM: Bypass DBG_DECRYPT API calls for unecrypted guest memory.
>
> Brijesh Singh (11):
> KVM: SVM: Add KVM_SEV SEND_START command
> KVM: SVM: Add KVM_SEND_UPDATE_DATA command
> KVM: SVM: Add KVM_SEV_SEND_FINISH command
> KVM: SVM: Add support for KVM_SEV_RECEIVE_START command
> KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command
> KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command
> KVM: x86: Add AMD SEV specific Hypercall3
> KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall
> KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl
> mm: x86: Invoke hypercall when page encryption status is changed
> KVM: x86: Introduce KVM_SET_PAGE_ENC_BITMAP ioctl
>
> .../virt/kvm/amd-memory-encryption.rst | 120 +++
> Documentation/virt/kvm/api.rst | 71 ++
> Documentation/virt/kvm/cpuid.rst | 5 +
> Documentation/virt/kvm/hypercalls.rst | 15 +
> Documentation/virt/kvm/msr.rst | 16 +
> arch/x86/include/asm/kvm_host.h | 7 +
> arch/x86/include/asm/kvm_para.h | 12 +
> arch/x86/include/asm/mem_encrypt.h | 11 +
> arch/x86/include/asm/paravirt.h | 10 +
> arch/x86/include/asm/paravirt_types.h | 2 +
> arch/x86/include/uapi/asm/kvm_para.h | 5 +
> arch/x86/kernel/kvm.c | 90 ++
> arch/x86/kernel/kvmclock.c | 12 +
> arch/x86/kernel/paravirt.c | 1 +
> arch/x86/kvm/svm/sev.c | 790 ++++++++++++++++++
> arch/x86/kvm/svm/svm.c | 21 +
> arch/x86/kvm/svm/svm.h | 9 +
> arch/x86/kvm/vmx/vmx.c | 1 +
> arch/x86/kvm/x86.c | 35 +
> arch/x86/mm/mem_encrypt.c | 68 +-
> arch/x86/mm/pat/set_memory.c | 7 +
> include/linux/efi.h | 1 +
> include/linux/psp-sev.h | 8 +-
> include/uapi/linux/kvm.h | 52 ++
> include/uapi/linux/kvm_para.h | 1 +
> 25 files changed, 1365 insertions(+), 5 deletions(-)
>
> --
> 2.17.1
>
