Ping Paolo & Sean,
Do you have any comment? Or do you think ITD virtualization is
appropriate to discuss at PUCK?
Thanks,
Zhao
On Sat, Feb 03, 2024 at 05:11:48PM +0800, Zhao Liu wrote:
> Date: Sat, 3 Feb 2024 17:11:48 +0800
> From: Zhao Liu <zhao1.liu@xxxxxxxxxxxxxxx>
> Subject: [RFC 00/26] Intel Thread Director Virtualization
> X-Mailer: git-send-email 2.34.1
>
> From: Zhao Liu <zhao1.liu@xxxxxxxxx>
>
> Hi list,
>
> This is our RFC to virtualize Intel Thread Director (ITD) feature for
> Guest, which is based on Ricardo's patch series about ITD related
> support in HFI driver ("[PATCH 0/9] thermal: intel: hfi: Prework for the
> virtualization of HFI" [1]).
>
> In short, the purpose of this patch set is to enable the ITD-based
> scheduling logic in Guest so that Guest can better schedule Guest tasks
> on Intel hybrid platforms.
>
> Currently, ITD is necessary for Windows VMs. Based on ITD virtualization
> support, the Windows 11 Guest could have significant performance
> improvement (for example, on i9-13900K, up to 14%+ improvement on
> 3DMARK).
>
> Our ITD virtualization is not bound to VMs' hybrid topology or vCPUs'
> CPU affinity. However, in our practice, the ITD scheduling optimization
> for win11 VMs works best when combined with hybrid topology and CPU
> affinity (this is related to the specific implementation of Win11
> scheduling). For more details, please see the Section.1.2 "About hybrid
> topology and vCPU pinning".
>
> To enable ITD related scheduling optimization in Win11 VM, some other
> thermal related support is also needed (HWP, CPPC), but we could emulate
> it with dummy value in the VMM (We'll also be sending out extra patches
> in the future for these).
>
> Welcome your feedback!
>
>
> 1. Background and Motivation
> ============================
>
> 1.1. Background
> ^^^^^^^^^^^^^^^
>
> We have the use case to run games in the client Windows VM as the cloud
> gaming solution.
>
> Gaming VMs are performance-sensitive VMs on Client, so that they usually
> have two characteristics to ensure interactivity and performance:
>
> i) There will be vCPUs equal to or close to the number of Host pCPUs.
>
> ii) The vCPUs of Gaming VM are often bound to the pCPUs to achieve
> exclusive resources and avoid the overhead of migration.
>
> In this case, Host can't provide effective scheduling for Guest, so we
> need to deliver more hardware-assisted scheduling capabilities to Guest
> to enhance Guest's scheduling.
>
> Windows 11 (and future Windows products) is heavily optimized for the
> Intel hybrid platform. To get the best performance, we need to
> virtualize hybrid scheduling features (HFI/ITD) for Windows Guest.
>
>
> 1.2. About hybrid topology and vCPU pinning
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Our ITD virtualization can support most vCPU topologies (except multiple
> packages/dies, see details in 3.5 Restrictions on Guest Topology), and
> can also support the case of non-pinning vCPUs (i.e. it can handle vCPU
> thread migration).
>
> The following is our performance measuremnt on an i9-13900K machine
> (2995Mhz, 24Cores, 32Thread(8+16) RAM: 14GB (16GB Physical)), with
> iGPU passthrough, running 3DMARK in Win11 Professional Guest:
>
>
> compared with smp topo case smp topo smp topo smp topo hybrid topo hybrid topo hybrid topo hybrid topo
> + affinity + ITD + ITD + affinity + ITD + ITD
> + affinity + affinity
> Time Spy - Overall 0.179% -0.250% 0.179% -0.107% 0.143% -0.179% -0.107%
> Graphics score 0.124% -0.249% 0.124% -0.083% 0.124% -0.166% -0.249%
> CPU score 0.916% -0.485% 1.149% -0.076% 0.722% -0.324% 11.915%
> Fire Strike Extreme - Overall 0.149% 0.000% 0.224% -1.021% -3.361% -1.319% -3.361%
> Graphics score 0.100% 0.050% 0.150% -1.376% -3.427% -1.676% -3.652%
> Physics score 5.060% 0.759% 0.518% -2.907% -10.914% -0.897% 14.638%
> Combined score 0.120% -0.179% 0.418% 0.060% -2.929% -0.179% -2.809%
> Fire Strike - Overall 0.350% -0.085% 0.193% -1.377% -1.365% -1.509% -1.787%
> Graphics score 0.256% -0.047% 0.210% -1.527% -1.376% -1.504% -2.320%
> Physics score 3.695% -2.180% 0.629% -1.581% -6.846% -1.444% 14.100%
> Combined score 0.415% -0.128% 0.128% -0.957% -1.052% -1.594% -0.957%
> CPU Profile Max Threads 1.836% 0.298% 1.786% -0.069% 1.545% 0.025% 9.472%
> 16 Threads 4.290% 0.989% 3.588% 0.595% 1.580% 0.848% 11.295%
> 8 Threads -22.632% -0.602% -23.167% -0.988% -1.345% -1.340% 8.648%
> 4 Threads -21.598% 0.449% -21.429% -0.817% 1.951% -0.832% 2.084%
> 2 Threads -12.912% -0.014% -12.006% -0.481% -0.609% -0.595% 1.161%
> 1 Threads -3.793% -0.137% -3.793% -0.495% -3.189% -0.495% 1.154%
>
>
> Based on the above result, we can find exposing only HFI/ITD to win11
> VMs without hybrid topology or CPU affinity (case "smp topo + ITD")
> won't hurt performance, but would also not get any performance
> improvement.
>
> Setting both hybrid topology and CPU affinity for ITD, then win11 VMs
> get significate performance improvement (up to 14%+, compared with the
> case setting smp topology without CPU affinity).
>
> Not only the numerical results of 3DMARK, but in practice, there is an
> significate improvement in the frame rate of the games.
>
> Also, the more powerful the machine, the more significate the
> performance gains!
>
> Therefore, the best practice for enabling ITD scheduling optimization
> is to set up both CPU affinity and hybrid topology for win11 Guest while
> enabling our ITD virtualization.
>
> Our earlier QEMU prototype RFC [2] presented the initial hybrid
> topology support for VMs. And currently our another proposal about
> "QOM topology" [3] has been raised in the QEMU community, which is the
> first step towards the hybrid topology implementation based on QOM
> approach.
>
>
> 2. Introduction of HFI and ITD
> ==============================
>
> Intel provides Hardware Feedback Interface (HFI) feature to allow
> hardware to provide guidance to the OS scheduler to perform optimal
> workload scheduling through a hardware feedback interface structure in
> memory [4]. This HFI structure is called HFI table.
>
> For now, the guidance includes performance and energy efficiency
> hints, and it could be update via thermal interrupt as the actual
> operating conditions of the processor change during run time.
>
> Intel Thread Director (ITD) feature extends the HFI to provide
> performance and energy efficiency data for advanced classes of
> instructions.
>
> Since ITD is an extension of HFI, our ITD virtualization also
> virtualizes the native HFI feature.
>
>
> 3. Dependencies of ITD
> ======================
>
> ITD is a thermal FEATURE that requires:
> * PTM (Package Thermal Management, alias, PTS)
> * HFI (Hardware Feedback Interface)
>
> In order to support the notification mechanism of ITD/HFI dynamic
> update, we also need to add thermal interrupt related support,
> including the following two features:
> * ACPI (Thermal Monitor and Software Controlled Clock Facilities)
> * TM (Thermal Monitor, alias, TM1/ACC)
>
> Therefore, we must also consider support for the emulation of all
> the above dependencies.
>
>
> 3.1. ACPI emulation
> ^^^^^^^^^^^^^^^^^^^
>
> For both ACPI, we can support it by emulating the RDMSR/WRMSR of the
> associated MSRs and adding the ability to inject thermal interrupts.
> But in fact, we don't really inject termal interrupts into Guest for
> the termal conditions corresponding to ACPI. Here the termal interrupt
> is prepared for the subsequent HFI/ITD.
>
>
> 3.2. TM emulation
> ^^^^^^^^^^^^^^^^^
>
> TM is a hardware feature and its CPUID bit only indicates the presence
> of the automatic thermal monitoring facilities. For TM, there's no
> interactive interface between OS and hardware, but its flag is one of
> the prerequisites for the OS to enable thermal interrupt.
>
> Thereby, as the support for TM, it is enough for us to expose its CPUID
> flag to Guest.
>
>
> 3.3. PTM emulation
> ^^^^^^^^^^^^^^^^^^
>
> PTM is a package-scope feature that includes package-level MSR and
> package-level thermal interrupt. Unfortunately, KVM currently only
> supports thread-scope MSR handling, and also doesn't care about the
> specific Guest's topology.
>
> But considering that our purpose of supporting PTM in KVM is to further
> support ITD, and the current platforms with ITD are all 1 package, so we
> emulate the MSRs of the package scope provided by PTM at the VM level.
>
> In this way, the VMM is required to set only one package topology for
> the PTM. In order to alleviate this limitation, we only expose the PTM
> feature bit to Guest when ITD needs to be supported.
>
>
> 3.4. HFI emulation
> ^^^^^^^^^^^^^^^^^^
>
> ITD is the extension of HFI, so both HFI and ITD depend on HFI table.
> HFI itself is used on the Host for power-related management control, so
> we should only expose HFI to Guest when we need to enable ITD.
>
> HFI also relies on PTM interrupt control, so it also has requirements
> for package topology, and we also emulate HFI (including ITD) at the VM
> level.
>
> In addition, because the HFI driver allocates HFI instances per die,
> this also affects HFI (and ITD) and must limit the Guest to only set one
> die.
>
>
> 3.5. Restrictions on Guest Topology
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Due to KVM's incomplete support for MSR topology and the requirement for
> HFI instance management in the kernel, PTM, HFI, and ITD limit the
> topology of the Guest (mainly restricting the topology types created on
> the VMM side).
>
> Therefore, we only expose PTM, HFI, and ITD to userspace when we need to
> support ITD. At the same time, considering that currently, ITD is only
> used on the client platform with 1 package and 1 die, such temporary
> restrictions will not have too much impact.
>
>
> 4. Overview of ITD (and HFI) virtualization
> ===========================================
>
> The main tasks of ITD (including HFI) virtualization are:
> * maintain a virtual HFI table for VM.
> * inject thermal interrupt when HFI table updates.
> * handle related MSRs' emulation and adjust HFI table based on MSR's
> control bits.
> * expose ITD/HFI configuration info in related CPUID leaves.
>
> The most important of these is the maintenance of the virtual HFI table.
> Although the HFI table should also be per package, since ITD/HFI related
> MSRs are treated as per VM in KVM, we also treat the virtual HFI table
> as per VM.
>
>
> 4.1. HFI table building
> ^^^^^^^^^^^^^^^^^^^^^^^
>
> HFI table contains a table header and many table entries. Each table
> entry is identified by an hfi table index, and each CPU corresponds to
> one of the hfi table indexes.
>
> ITD and HFI features both depend on the HFI table, but their HFI table
> are a little different. The HFI table provided by the ITD feature has
> more classes (in terms of more columns in the table) than the HFI table
> of native HFI feature.
>
> The virtual HFI table in KVM is built based on the actual HFI table,
> which is maintained by HFI instance in HFI driver. We extract the HFI
> data of the pCPUs, which vCPUs are running on, to form a virtual HFI
> table.
>
>
> 4.2. HFI table index
> ^^^^^^^^^^^^^^^^^^^^
>
> There are many entries in the HFI table, and the vCPU will be assigned
> an HFI table index to specify the entry it maps. KVM will fill the
> pCPU's HFI data (the pCPU that vCPU is running on) into the entry
> corresponding to the HFI table index of the vCPU in the vcitual HFI
> table.
>
> This index is set by VMM in CPUID.
>
>
> 4.3. HFI table updating
> ^^^^^^^^^^^^^^^^^^^^^^^
>
> On some platforms, the HFI table will be dynamically updated with
> thermal interrupts. In order to update the virtual HFI table in time, we
> added the per-VM notifier to the HFI driver to notify KVM to update the
> virtual HFI table for the VM, and then inject thermal interrupt into the
> VM to notify the Guest.
>
> There is another case that needs to update the virtual HFI table, that
> is, when the vCPU is migrated, the pCPU where it is located is changed,
> and the corresponding virtual HFI data should also be updated to the new
> pCPU's data. In this case, in order to reduce overhead, we can only
> update the data of a single vPCU without traversing the entire virtual
> HFI table.
>
>
> 5. Patch Summary
> ================
>
> Patch 01-03: Prepare the bit definition, the hfi helpers and hfi data
> structures that KVM needs.
> Patch 04-05: Add the sched_out arch hook and reset the classification
> history at sched_in()/schedu_out().
> Patch 06-10: Add emulations of ACPI, TM and PTM, mainly about CPUID and
> related MSRs.
> Patch 11-20: Add the emulation support for HFI, including maintaining
> the HFI table for VM.
> Patch 21-23: Add the emulation support for ITD, including extending HFI
> to ITD and passing through the classification MSRs.
> Patch 24-25: Add HRESET emulation support, which is also used by IPC
> classes feature.
> Patch 26: Add the brief doc about the per-VM lock - pkg_therm_lock.
>
>
> 6. References
> =============
>
> [1]: [PATCH 0/9] thermal: intel: hfi: Prework for the virtualization of HFI
> https://lore.kernel.org/lkml/20240203040515.23947-1-ricardo.neri-calderon@xxxxxxxxxxxxxxx/
> [2]: [RFC 00/52] Introduce hybrid CPU topology,
> https://lore.kernel.org/qemu-devel/20230213095035.158240-1-zhao1.liu@xxxxxxxxxxxxxxx/
> [3]: [RFC 00/41] qom-topo: Abstract Everything about CPU Topology,
> https://lore.kernel.org/qemu-devel/20231130144203.2307629-1-zhao1.liu@xxxxxxxxxxxxxxx/
> [4]: SDM, vol. 3B, section 15.6 HARDWARE FEEDBACK INTERFACE AND INTEL
> THREAD DIRECTOR
>
>
> Thanks and Best Regards,
> Zhao
> ---
> Zhao Liu (17):
> thermal: Add bit definition for x86 thermal related MSRs
> KVM: Add kvm_arch_sched_out() hook
> KVM: x86: Reset hardware history at vCPU's sched_in/out
> KVM: VMX: Add helpers to handle the writes to MSR's R/O and R/WC0 bits
> KVM: x86: cpuid: Define CPUID 0x06.eax by kvm_cpu_cap_mask()
> KVM: VMX: Introduce HFI description structure
> KVM: VMX: Introduce HFI table index for vCPU
> KVM: x86: Introduce the HFI dynamic update request and kvm_x86_ops
> KVM: VMX: Allow to inject thermal interrupt without HFI update
> KVM: VMX: Emulate HFI related bits in package thermal MSRs
> KVM: VMX: Emulate the MSRs of HFI feature
> KVM: x86: Expose HFI feature bit and HFI info in CPUID
> KVM: VMX: Extend HFI table and MSR emulation to support ITD
> KVM: VMX: Pass through ITD classification related MSRs to Guest
> KVM: x86: Expose ITD feature bit and related info in CPUID
> KVM: VMX: Emulate the MSR of HRESET feature
> Documentation: KVM: Add description of pkg_therm_lock
>
> Zhuocheng Ding (9):
> thermal: intel: hfi: Add helpers to build HFI/ITD structures
> thermal: intel: hfi: Add HFI notifier helpers to notify HFI update
> KVM: VMX: Emulate ACPI (CPUID.0x01.edx[bit 22]) feature
> KVM: x86: Expose TM/ACC (CPUID.0x01.edx[bit 29]) feature bit to VM
> KVM: VMX: Emulate PTM/PTS (CPUID.0x06.eax[bit 6]) feature
> KVM: VMX: Support virtual HFI table for VM
> KVM: VMX: Sync update of Host HFI table to Guest
> KVM: VMX: Update HFI table when vCPU migrates
> KVM: x86: Expose HRESET feature's CPUID to Guest
>
> Documentation/virt/kvm/locking.rst | 13 +-
> arch/arm64/include/asm/kvm_host.h | 1 +
> arch/mips/include/asm/kvm_host.h | 1 +
> arch/powerpc/include/asm/kvm_host.h | 1 +
> arch/riscv/include/asm/kvm_host.h | 1 +
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/x86/include/asm/hfi.h | 28 ++
> arch/x86/include/asm/kvm-x86-ops.h | 3 +-
> arch/x86/include/asm/kvm_host.h | 2 +
> arch/x86/include/asm/msr-index.h | 54 +-
> arch/x86/kvm/cpuid.c | 201 +++++++-
> arch/x86/kvm/irq.h | 1 +
> arch/x86/kvm/lapic.c | 9 +
> arch/x86/kvm/svm/svm.c | 8 +
> arch/x86/kvm/vmx/vmx.c | 751 +++++++++++++++++++++++++++-
> arch/x86/kvm/vmx/vmx.h | 79 ++-
> arch/x86/kvm/x86.c | 18 +
> drivers/thermal/intel/intel_hfi.c | 212 +++++++-
> drivers/thermal/intel/therm_throt.c | 1 -
> include/linux/kvm_host.h | 1 +
> virt/kvm/kvm_main.c | 1 +
> 21 files changed, 1343 insertions(+), 44 deletions(-)
>
> --
> 2.34.1
>
