FWIW, I really appreciate this mail which explains to unenlightened people like me what this is all about. Especially the Acronyms, Flow, Specs pointers etc. I wish I could see this type of writeups in all patchsets' 0th messages, leading in the reader into the topic while not expecting the latter to actually *know* all those things because, d0h, it is obvious. You can read my mind, right? :-) So thanks for taking the time - it is very helpful! On Tue, Feb 18, 2025 at 10:09:47PM +1100, Alexey Kardashevskiy wrote: > Here are some patches to enable SEV-TIO on AMD Turin. It's been a while > and got quiet and I kept fixing my tree and wondering if I am going in > the right direction. > > SEV-TIO allow a guest to establish trust in a device that supports TEE > Device Interface Security Protocol (TDISP, defined in PCIe r6.0+) and > then interact with the device via private memory. > > These include both guest and host support. QEMU also requires changes. > This is more to show what it takes on AMD EPYC to pass through TDISP > devices, hence "RFC". > > Components affected: > KVM > IOMMUFD > CCP (AMD) > SEV-GUEST (AMD) > > New components: > PCI IDE > PCI TSM > VIRT CoCo TSM > VIRT CoCo TSM-HOST > VIRT CoCo TSM-GUEST > > > This is based on a merge of Lukas'es CMA and 1 week old upstream + some of Dan's patches: > > https://github.com/aik/linux/tree/tsm > https://github.com/aik/qemu/tree/tsm > > Not using "[PATCH 03/11] coco/tsm: Introduce a class device for TEE Security Managers" > yet as may be (may be) my approach makes sense too. Tried to stick to the terminology. > I have done some changes on top of that, these are on github, not posting here as > I expect those to be addressed in that thread: > https://lore.kernel.org/linux-coco/173343739517.1074769.13134786548545925484.stgit@xxxxxxxxxxxxxxxxxxxxxxxxx/T/ > > > SEV TIO spec: > https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/specifications/58271_0_70.pdf > Whitepaper: > https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/sev-tio-whitepaper.pdf > > > Acronyms: > > TEE - Trusted Execution Environments, a concept of managing trust between the host > and devices > TSM - TEE Security Manager (TSM), an entity which ensures security on the host > PSP - AMD platform secure processor (also "ASP", "AMD-SP"), acts as TSM on AMD. > SEV TIO - the TIO protocol implemented by the PSP and used by the host > GHCB - guest/host communication block - a protocol for guest-to-host communication > via a shared page > TDISP - TEE Device Interface Security Protocol (PCIe). > > > Flow: > > - Boot host OS, load CCP and PCI TSM (they will load TSM-HOST too) > - PCI TSM creates sysfs nodes in "coco/tsm: Add tsm and tsm-host modules" for all TDISP-capable devices > - Enable IDE via "echo 0 > /sys/bus/pci/devices/0000:e1:00.0/tsm-dev/tdev:0000:e1:00.0/tsm_dev_connect" > - Examine certificates/measurements/status via sysfs > > - run an SNP VM _without_ VFIO PCI device, wait till it is booted > - hotplug a TDISP-capable PCI function, IOMMUFD must be used (not a VFIO container) > - QEMU pins all guest memory via IOMMUFD map-from-fd ioctl() > - the VM detects a TDISP-capable device, creates sysfs nodes in "coco/tsm: Add tsm-guest module" > - the VM loads the device driver which goes as usual till enabling bus master (for convinience) > - TSM-GUEST modules listens for bus master event (hacked in "pci: Add BUS_NOTIFY_PCI_BUS_MASTER event") > - TSM-GUEST requests TDI ("trusted PCI VF") info, traps into QEMU > - QEMU binds the VF to the Coco VM in the secure fw (AMD PSP) via IOMMUFD ioctl > - QEMU reads certificates/measurements/interface report from the hosts sysfs and writes to the guest memory > - the guest receives all the data, examines it (not in this series though) > - the guest enables secure DMA and MMIO by calling GHCB which traps into QEMU > - QEMU calls IOMMUFD ioctl to enable secure DMA and MMIO > - the guest can now stop sharing memory for DMA (and expect DMA to encrypted memory to work) and > start accessing validated MMIO with Cbit set. > > > > Assumptions > > This requires hotpligging into the VM vs passing the device via the command line as > VFIO maps all guest memory as the device init step which is too soon as > SNP LAUNCH UPDATE happens later and will fail if VFIO maps private memory before that. > > This requires the BME hack as MMIO and BusMaster enable bits cannot be 0 after MMIO > validation is done and there are moments in the guest OS booting process when this > appens. > > SVSM could help addressing these (not implemented). > > QEMU advertises TEE-IO capability to the VM. An additional x-tio flag is added to > vfio-pci. > > Trying to avoid the device driver modification as much as possible at > the moment as my test devices already exist in non-TDISP form and need to work without > modification. Arguably this may not be always the case. > > > TODOs > > Deal with PCI reset. Hot unplug+plug? Power states too. > Actually collaborate with CMA. > Other tons of things. > > > The previous conversation is here: > https://lore.kernel.org/r/20240823132137.336874-1-aik@xxxxxxx > > > Changes: > v2: > * redid the whole thing pretty much > * RMPUPDATE API for QEMU > * switched to IOMMUFD > * mapping guest memory via IOMMUFD map-from-fd > * marking resouces as validated > * more modules > * moved tons to the userspace (QEMU), such as TDI bind and GHCB guest requests > > > Sean, get_maintainer.pl produced more than 100 emails for the entire > patchset, should I have posted them all anyway? > > Please comment. Thanks. > > > > Alexey Kardashevskiy (22): > pci/doe: Define protocol types and make those public > PCI/IDE: Fixes to make it work on AMD SNP-SEV > PCI/IDE: Init IDs on all IDE streams beforehand > iommu/amd: Report SEV-TIO support > crypto: ccp: Enable SEV-TIO feature in the PSP when supported > KVM: X86: Define tsm_get_vmid > coco/tsm: Add tsm and tsm-host modules > pci/tsm: Add PCI driver for TSM > crypto/ccp: Implement SEV TIO firmware interface > KVM: SVM: Add uAPI to change RMP for MMIO > KVM: SEV: Add TIO VMGEXIT > iommufd: Allow mapping from guest_memfd > iommufd: amd-iommu: Add vdevice support > iommufd: Add TIO calls > KVM: X86: Handle private MMIO as shared > coco/tsm: Add tsm-guest module > resource: Mark encrypted MMIO resource on validation > coco/sev-guest: Implement the guest support for SEV TIO > RFC: pci: Add BUS_NOTIFY_PCI_BUS_MASTER event > sev-guest: Stop changing encrypted page state for TDISP devices > pci: Allow encrypted MMIO mapping via sysfs > pci: Define pci_iomap_range_encrypted > > drivers/crypto/ccp/Makefile | 13 + > drivers/pci/Makefile | 3 + > drivers/virt/coco/Makefile | 2 + > drivers/virt/coco/guest/Makefile | 3 + > drivers/virt/coco/host/Makefile | 6 + > drivers/virt/coco/sev-guest/Makefile | 2 +- > arch/x86/include/asm/kvm-x86-ops.h | 1 + > arch/x86/include/asm/kvm_host.h | 2 + > arch/x86/include/asm/sev.h | 31 + > arch/x86/include/uapi/asm/kvm.h | 11 + > arch/x86/include/uapi/asm/svm.h | 2 + > drivers/crypto/ccp/sev-dev-tio.h | 111 ++ > drivers/crypto/ccp/sev-dev.h | 19 + > drivers/iommu/amd/amd_iommu_types.h | 3 + > drivers/iommu/iommufd/iommufd_private.h | 3 + > include/asm-generic/pci_iomap.h | 4 + > include/linux/amd-iommu.h | 2 + > include/linux/device.h | 4 + > include/linux/device/bus.h | 3 + > include/linux/dma-direct.h | 8 + > include/linux/ioport.h | 2 + > include/linux/kvm_host.h | 2 + > include/linux/pci-doe.h | 4 + > include/linux/pci-ide.h | 19 +- > include/linux/pci.h | 2 +- > include/linux/psp-sev.h | 61 +- > include/linux/swiotlb.h | 8 + > include/linux/tsm.h | 315 ++++ > include/uapi/linux/iommufd.h | 26 + > include/uapi/linux/kvm.h | 24 + > include/uapi/linux/pci_regs.h | 5 +- > include/uapi/linux/psp-sev.h | 6 +- > include/uapi/linux/sev-guest.h | 39 + > arch/x86/coco/sev/core.c | 19 +- > arch/x86/kvm/mmu/mmu.c | 6 +- > arch/x86/kvm/svm/sev.c | 205 +++ > arch/x86/kvm/svm/svm.c | 12 + > arch/x86/mm/ioremap.c | 2 + > arch/x86/mm/mem_encrypt.c | 6 + > arch/x86/virt/svm/sev.c | 34 +- > drivers/crypto/ccp/sev-dev-tio.c | 1664 ++++++++++++++++++++ > drivers/crypto/ccp/sev-dev-tsm.c | 709 +++++++++ > drivers/crypto/ccp/sev-dev.c | 94 +- > drivers/iommu/amd/init.c | 9 + > drivers/iommu/amd/iommu.c | 60 +- > drivers/iommu/iommufd/main.c | 6 + > drivers/iommu/iommufd/pages.c | 88 +- > drivers/iommu/iommufd/viommu.c | 112 ++ > drivers/pci/doe.c | 2 - > drivers/pci/ide.c | 103 +- > drivers/pci/iomap.c | 24 + > drivers/pci/mmap.c | 11 +- > drivers/pci/pci-sysfs.c | 27 +- > drivers/pci/pci.c | 3 + > drivers/pci/proc.c | 2 +- > drivers/pci/tsm.c | 233 +++ > drivers/virt/coco/guest/tsm-guest.c | 326 ++++ > drivers/virt/coco/host/tsm-host.c | 551 +++++++ > drivers/virt/coco/sev-guest/sev_guest.c | 10 + > drivers/virt/coco/sev-guest/sev_guest_tio.c | 738 +++++++++ > drivers/virt/coco/tsm.c | 638 ++++++++ > kernel/resource.c | 48 + > virt/kvm/kvm_main.c | 6 + > Documentation/virt/coco/tsm.rst | 132 ++ > drivers/crypto/ccp/Kconfig | 2 + > drivers/pci/Kconfig | 15 + > drivers/virt/coco/Kconfig | 14 + > drivers/virt/coco/guest/Kconfig | 3 + > drivers/virt/coco/host/Kconfig | 6 + > drivers/virt/coco/sev-guest/Kconfig | 1 + > 70 files changed, 6614 insertions(+), 53 deletions(-) > create mode 100644 drivers/virt/coco/host/Makefile > create mode 100644 drivers/crypto/ccp/sev-dev-tio.h > create mode 100644 drivers/crypto/ccp/sev-dev-tio.c > create mode 100644 drivers/crypto/ccp/sev-dev-tsm.c > create mode 100644 drivers/pci/tsm.c > create mode 100644 drivers/virt/coco/guest/tsm-guest.c > create mode 100644 drivers/virt/coco/host/tsm-host.c > create mode 100644 drivers/virt/coco/sev-guest/sev_guest_tio.c > create mode 100644 drivers/virt/coco/tsm.c > create mode 100644 Documentation/virt/coco/tsm.rst > create mode 100644 drivers/virt/coco/host/Kconfig > > -- > 2.47.1 > -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
