What's changed from v2: a) Detailed feature description b) refine description in "Address translation in virtual SVM" b) "Terms" is added Content =============================================== 1. Feature description 2. Why use it? 3. How to enable it 4. How to test 5. Terms Details =============================================== 1. Feature description Shared virtual memory(SVM) is to let application program share its virtual address with SVM capable devices. Shared virtual memory details: a) SVM feature requires ATS/PRQ/PASID support on both device side and IOMMU side. b) SVM capable devices could send DMA requests with PASID, the address in the request would be a virtual address within a program's virtual address space. c) IOMMU would use first level page table to translate the address in the request. d) First level page table is a HVA->HPA mapping on bare metal. Shared Virtual Memory feature in pass-through scenarios is actually SVM virtualization. It is to let application programs(running in guest)share their virtual address with assigned device(e.g. graphics processors or accelerators). In virtualization, SVM would be: a) Require a vIOMMU exposed to guest b) Assigned SVM capable device could send DMA requests with PASID, the address in the request would be a virtual address within a guest program's virtual address space(GVA). c) Physical IOMMU needs to do GVA->GPA->HPA translation. Nested mode would be enabled, first level page table would achieve GVA->GPA mapping, while second level page table would achieve GPA->HPA translation. For more SVM detail, you may want refer to section 2.5.1.1 of Intel VT-d spec and section 5.6 of OpenCL spec. For details about SVM address translation, pls refer to section 3 of Intel VT-d spec. It's also welcomed to discuss directly in this thread. Link to related specs: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf https://www.khronos.org/registry/cl/specs/opencl-2.0.pdf 2. Why use it? It is common to pass-through devices to guest and expect to achieve as much similar performance as it is on host. With this feature enabled, the application programs in guest would be able to share data-structures with assigned devices without unnecessary overheads. 3. How to enable it As mentioned above, SVM virtualization requires a vIOMMU exposed to guest. Since there is an existing IOMMU emulator in host user space(QEMU), it is more acceptable to extend the IOMMU emulator to support SVM for assigned devices. So far, the vIOMMU exposed to guest is only for emulated devices. In this design, it would focus on virtual SVM for assigned devices. Virtual IOVA and virtual interrupt remapping will not be included here. The enabling work would include the following items. a) IOMMU Register Access Emulation Already existed in QEMU, need some extensions to support SVM. e.g. support page request service related registers(PQA_REG). b) vIOMMU Capability Report SVM related capabilities(PASID,PRS,DT,PT,ECS etc.) in ex-capability register and cache mode, DWD, DRD in capability register. c) QI Handling Emulation Already existed in QEMU, need to shadow the QIs related to assigned devices to physical IOMMU. i. ex-context entry cache invalidation(nested mode setting, guest PASID table pointer shadowing) ii. 1st level translation cache invalidation iii. Response for recoverable faults d) Address translation in virtual SVM In virtualization, for requests with PASID from assigned device, the address translation would be subjected to first level page table and then second level page table, which is named nested mode. Extended context mode should be supported on hardware. DMA remapping in SVM virtualization would be: i. For requests with PASID, the related extended context entry should have the NESTE bit set. ii. Guest PASID table pointer should be shadowed to host IOMMU driver. The PASID table pointer field in extended context entry would be a GPA as nested mode is on. First level page table would be maintained by guest IOMMU driver. Second level page table would be maintained by host IOMMU driver. e) Recoverable Address Translation Faults Handling Emulation It is serviced by page request when device support PRS. For assigned devices, host IOMMU driver would get page requests from pIOMMU. Here, we need a mechanism to drain the page requests from devices which are assigned to a guest. In this design it would be done through VFIO. Page request descriptors would be propagated to user space and then exposed to guest IOMMU driver. This requires following support: i. a mechanism to notify vIOMMU emulator to fetch PRQ descriptor ii. a notify framework in QEMU to signal the PRQ descriptor fetching when notified by pIOMMU f) Non-Recoverable Address Translation Handling Emulation The non-recoverable fault propagation is similar to recoverable faults. In this design it would propagate fault data to user space(QEMU) through VFIO. vIOMMU emulator then emulate the fault. Either fill data to vIOMMU fault record registers or fill the data to memory-resident fault log region. Depends on the fault reporting type.(primary fault logging or advanced fault logging) g) SVM Virtualization Architecture ********************************************************************** Guest +------------------+ +->| vIOMMU driver | | +------+---------+-+ | | | +----+(1) |(2) |(3) | | | *****************************************|****|*********|************* Host User | v v Space +--+---------------------+ | Qemu vIOMMU | +--+----+----+----+----+-+ | | | | | *****************************************|****|****|****|****|******** Host Kernel |(1) |(2) |(4) |(5) |(6) Space | | | | | +-----------------------+ +---+----+----+----+----+---+ | IOMMU Fault | | | | | | | | | +--------------------+-------+---+ | | | | | | | | | | | | | | | | +-----------------+-------+--------+ | | | | | | | | | | | | | | | | IOMMU | | VFIO | | | | | | | Driver | | | | | | | | | +--------+-------+-------------+ | | | | | | | | | | | | | | | | +-----+-------+------------------+ | | | | | | | | | | | | | | | | +--+-------|-----------------------+ | | | | | | | | | | +--+--+--------+--+--+--+ +---------------------------+ | | | | | ********|**|********|**|**|******************************************* HW | | | | | | v v v v +-+--------------------+ | pIOMMU | +----------------------+ ********************************************************************** (1)Fault reporting, include recoverable and un-recoverable faults (2)PRQ response (3)Translation cache invalidation(QI) (4)Set nested mode in pIOMMU ex-context entry (5)Shadow gPASID table pointer to pIOMMU ex-context entry (6)Cache invalidation for 1st level translation <if the diagram is disordered, you may want to paste it to a Linux email client> 4. How to test Test would be done with devices which has SVM capability. Hereby, Intel i915 GPU would be chosen to do the verification. Intel provides three tools from for SVM verification. They are: i) intel-gpu-tools/tests/gem_svm_sanity ii) intel-gpu-tools/tests/gem_svm_fault iii) intel-gpu-tools/tests/gem_svm_storedw_loop_render The following scenarios would have to be covered: a) Test case 1 - SVM usage in host i) Requires a physical machine which has at least one SVM capable device. ii) Run Test Tools in host. iii) Expect: with vSVM enabled, it shouldn't affect SVM usage in host b) Test case 2 - SVM usage in guest i) Requires a physical machine which has at least one SVM capable device. ii) Create a guest, and assign a SVM capable device to it. iii) Run Test Tools in the guest. iV) Expect: with vSVM enabled and device assigned, guest should be able to use SVM with the assigned device c) Test case 3 - SVM usage in multi-guests scenario i) Requires a physical machine which has at least two SVM capable devices. ii) Create two guests, and assign a SVM capable device to each of them. iii) Run Test Tools on both of the two guests. iV) Expect: multi-guest should be able to use SVM with its assigned devices without affect each other d) Test case 4 - SVM usage in host/guest scenario i) Requires a physical machine which has at least two SVM capable devices. ii) Create a guest, and assign a SVM capable device to the guest. iii) Run Test Tools on both of the host and the guest. iV) Expect: host and guest shouldn't affect each other 5. Terms: SVM: Shared Virtual Memory CSR: Means the IOMMU registers in this slide IOVA: IO Virtual Address PRQ: Page Request vIOMMU: Virtual IOMMU emulated by QEMU FLPT: First Level Page Table SLPT: Second Level Page Table QI: Queued Invalidation, a mechanism used to invalidate cache in VTd PASID: Process Address Space ID Thanks, Best Wishes, Yi Liu -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
