We wanted to check whether there are KVM regressions with the v5 FRED patch set, and had a round of Kselftest on KVM guests. Following are the results on X86-64. +--------------------------------+-------+-------+-------+-------+ | Config | Pass | Fail | Skip | Hang | +--------------------------------+-------+-------+-------+-------+ | the 7th Intel(R) Core(TM) CPU | | | | | | | host 6.3.0-rc1+ w/ FRED | 2720 | 403 | 689 | 9 | | patch set FRED disabled | | | | | | | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | the 7th Intel(R) Core(TM) CPU | | | | | | | host 6.3.0-rc1+ w/ FRED | 2720 | 403 | 689 | 9 | | patch set | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o | | | | | | FRED model | host 6.3.0-rc1+ | | 1403 | 277 | 2127 | 14 | | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o | | | | | | FRED model | host 6.3.0-rc1+ | | | | | | w/ FRED patch set | guest | 1403 | 277 | 2127 | 14 | | 6.3.0-rc1+ w/ FRED patch set | | | | | | FRED disabled | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o | | | | | | FRED model | host 6.3.0-rc1+ | 1403 | 277 | 2127 | 14 | | w/ FRED patch set | guest | | | | | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model + host 6.3.0-rc1+ | | 1404 | 276 | 2127 | 14 | | guest 6.3.0-rc1+ w/ FRED patch | | | | | | set | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | 1404 | 276 | 2127 | 14 | | w/ FRED patch set FRED | | | | | | disabled | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | | | | | | w/ FRED patch set FRED | 1404 | 276 | 2127 | 14 | | disabled | guest 6.3.0-rc1+ w/ | | | | | | FRED patch set | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | 1404 | 276 | 2127 | 14 | | w/ FRED patch set | guest | | | | | | 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | | | | | | w/ FRED patch set | guest | 1404 | 276 | 2127 | 14 | | 6.3.0-rc1+ w/ FRED patch set | | | | | | FRED disabled | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | 1404 | 276 | 2127 | 14 | | w/ FRED patch set | guest | | | | | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------+-------+-------+-------+-------+ The Simics FRED model has one more passed case, which is x86:test_vsyscall_32, due to https://lore.kernel.org/lkml/20230311084824.2340-1-xin3.li@xxxxxxxxx/. Thanks --Shan On Sat, 2023-03-11 at 09:58 +0000, Kang, Shan wrote: > We tested the v5 FRED patch set on the Intel Simics® Simulator and a machine > with a 7th Intel(R) Core(TM) CPU. > > Following are the Kselftest results on X86-64. > +--------------------------------------------+-------+-------+-------+-------+ > > Config | Pass | Fail | Skip | Hang | > +--------------------------------------------+-------+-------+-------+-------+ > > the 7th Intel(R) Core(TM) CPU | 3078 | 458 | 734 | 5 | > > 6.3.0-rc1+ | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > the 7th Intel(R) Core(TM) CPU | 3078 | 458 | 734 | 5 | > > 6.3.0-rc1+ w/ FRED patch set | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > Intel Simics® Simulator w/o FRED model | 1888 | 271 | 2105 | 11 | > > 6.3.0-rc1+ | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > Intel Simics® Simulator w/o FRED model | 1888 | 271 | 2105 | 11 | > > 6.3.0-rc1+ w/ FRED patch set | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > Intel Simics® Simulator w/ FRED model | 1889 | 270 | 2105 | 11 | > > 6.3.0-rc1+ | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > Intel Simics® Simulator w/ FRED model | 1889 | 270 | 2105 | 11 | > > 6.3.0-rc1+ w/ FRED patch set FRED disabled | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > Intel Simics® Simulator w/ FRED model | 1888 | 270 | 2105 | 12 | > > 6.3.0-rc1+ w/ FRED patch set | | | | | > +--------------------------------------------+-------+-------+-------+-------+ > > The following issues are seen in this round of test. > +----------------+----------------+----------------+----------------+ > > | x86:test_ | bpf:test_progs | x86:sysret | > > | vsyscall_32 | | _rip_64 | > +----------------+----------------+----------------+----------------+ > > the 7th | | | | > > Intel(R) | FAIL | FAIL | PASS | > > Core(TM) CPU | | | | > > 6.3.0-rc1+ | | | | > +----------------+----------------+----------------+----------------+ > > the 7th | | | | > > Intel(R) | | | | > > Core(TM) CPU | FAIL | FAIL | PASS | > > 6.3.0-rc1+ w/ | | | | > > FRED patch set | | | | > +----------------+----------------+----------------+----------------+ > > Intel Simics® | | | | > > Simulator w/o | FAIL | FAIL | PASS | > > FRED model | | | | > > 6.3.0-rc1+ | | | | > +----------------+----------------+----------------+----------------+ > > Intel Simics® | | | | > > Simulator w/o | | | | > > FRED model | FAIL | FAIL | PASS | > > 6.3.0-rc1+ w/ | | | | > > FRED patch set | | | | > +----------------+----------------+----------------+----------------+ > > Intel Simics® | | | | > > Simulator w/ | PASS | FAIL | PASS | > > FRED model | | | | > > 6.3.0-rc1+ | | | | > +----------------+----------------+----------------+----------------+ > > Intel Simics® | | | | > > Simulator w/ | | | | > > FRED model | PASS | FAIL | PASS | > > 6.3.0-rc1+ w/ | | | | > > FRED patch set | | | | > > FRED disabled | | | | > +----------------+----------------+----------------+----------------+ > > Intel Simics® | | | | > > Simulator w/ | | | | > > FRED model | PASS | HANG | FAIL | > > 6.3.0-rc1+ w/ | | | | > > FRED patch set | | | | > +----------------+----------------+----------------+----------------+ > > The test "x86:sysret_rip_64" is NOT a valid test on FRED, and there is a fix > from Ammar Faizi after we discussed it in the LKML. > > The test "bpf:test_progs" is still in investigation. > > The "x86:test_vsyscall_32" is a regression since the v3 FRED patch set. > > Thanks > --Shan > > On Mon, 2023-03-06 at 18:39 -0800, Xin Li wrote: > > This patch set enables FRED for x86-64. > > > > The Intel flexible return and event delivery (FRED) architecture defines > > simple > > new transitions that change privilege level (ring transitions). The FRED > > architecture was designed with the following goals: > > 1) Improve overall performance and response time by replacing event delivery > > through the interrupt descriptor table (IDT event delivery) and event return > > by > > the IRET instruction with lower latency transitions. > > 2) Improve software robustness by ensuring that event delivery establishes > > the > > full supervisor context and that event return establishes the full user > > context. > > > > The new transitions defined by the FRED architecture are FRED event delivery > > and, > > for returning from events, two FRED return instructions. FRED event delivery > > can > > effect a transition from ring 3 to ring 0, but it is used also to deliver > > events > > incident to ring 0. One FRED instruction (ERETU) effects a return from ring > > 0 > > to > > ring 3, while the other (ERETS) returns while remaining in ring 0. > > > > Search for the latest FRED spec in most search engines with this search > > pattern: > > > > site:intel.com FRED (flexible return and event delivery) specification > > > > As of now there is no publicly avaiable CPU supporting FRED, thus the Intel > > Simics® Simulator is used as software development and testing vehicles. And > > it can be downloaded from: > > > > https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html > > > > To enable FRED, the Simics package 8112 QSP-CPU needs to be installed with > > CPU > > model configured as: > > $cpu_comp_class = "x86-experimental-fred" > > > > Longer term, we should refactor common code shared by FRED and IDT into > > common > > shared files, and contain IDT code using a new config CONFIG_X86_IDT. > > > > Changes since v4: > > * Rebased against v6.3-rc1. > > * Do NOT use the term "injection", which in the KVM context means to > > reinject an event into the guest (Sean Christopherson). > > * Add the explanation of why to execute "int $2" to invoke the NMI handler > > in NMI caused VM exits (Sean Christopherson). > > * Use cs/ss instead of csx/ssx when initializing the pt_regs structure > > for calling external_interrupt(), otherwise it breaks i386 build. > > > > Changes since v3: > > * Call external_interrupt() to handle IRQ in IRQ caused VM exits. > > * Execute "int $2" to handle NMI in NMI caused VM exits. > > * Rename csl/ssl of the pt_regs structure to csx/ssx (x for extended) > > (Andrew Cooper). > > > > Changes since v2: > > * Improve comments for changes in arch/x86/include/asm/idtentry.h. > > > > Changes since v1: > > * call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel > > handler (Peter Zijlstra). > > * Initialize a FRED exception handler to fred_bad_event() instead of NULL > > if no FRED handler defined for an exception vector (Peter Zijlstra). > > * Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}() > > down into individual FRED exception handlers, instead of in the dispatch > > framework (Peter Zijlstra). > > > > > > H. Peter Anvin (Intel) (24): > > x86/traps: let common_interrupt() handle IRQ_MOVE_CLEANUP_VECTOR > > x86/traps: add a system interrupt table for system interrupt dispatch > > x86/traps: add external_interrupt() to dispatch external interrupts > > x86/cpufeature: add the cpu feature bit for FRED > > x86/opcode: add ERETU, ERETS instructions to x86-opcode-map > > x86/objtool: teach objtool about ERETU and ERETS > > x86/cpu: add X86_CR4_FRED macro > > x86/fred: add Kconfig option for FRED (CONFIG_X86_FRED) > > x86/fred: if CONFIG_X86_FRED is disabled, disable FRED support > > x86/cpu: add MSR numbers for FRED configuration > > x86/fred: header file with FRED definitions > > x86/fred: make unions for the cs and ss fields in struct pt_regs > > x86/fred: reserve space for the FRED stack frame > > x86/fred: add a page fault entry stub for FRED > > x86/fred: add a debug fault entry stub for FRED > > x86/fred: add a NMI entry stub for FRED > > x86/fred: FRED entry/exit and dispatch code > > x86/fred: FRED initialization code > > x86/fred: update MSR_IA32_FRED_RSP0 during task switch > > x86/fred: let ret_from_fork() jmp to fred_exit_user when FRED is > > enabled > > x86/fred: disallow the swapgs instruction when FRED is enabled > > x86/fred: no ESPFIX needed when FRED is enabled > > x86/fred: allow single-step trap and NMI when starting a new thread > > x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1f > > > > Xin Li (10): > > x86/traps: add install_system_interrupt_handler() > > x86/traps: export external_interrupt() for VMX IRQ reinjection > > x86/fred: header file for event types > > x86/fred: add a machine check entry stub for FRED > > x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user > > x86/ia32: do not modify the DPL bits for a null selector > > x86/fred: allow dynamic stack frame size > > x86/fred: disable FRED by default in its early stage > > KVM: x86/vmx: call external_interrupt() to handle IRQ in IRQ caused VM > > exits > > KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits > > when FRED is enabled > > > > .../admin-guide/kernel-parameters.txt | 4 + > > arch/x86/Kconfig | 9 + > > arch/x86/entry/Makefile | 5 +- > > arch/x86/entry/entry_32.S | 2 +- > > arch/x86/entry/entry_64.S | 5 + > > arch/x86/entry/entry_64_fred.S | 59 +++++ > > arch/x86/entry/entry_fred.c | 234 ++++++++++++++++++ > > arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- > > arch/x86/include/asm/cpufeatures.h | 1 + > > arch/x86/include/asm/disabled-features.h | 8 +- > > arch/x86/include/asm/entry-common.h | 3 + > > arch/x86/include/asm/event-type.h | 17 ++ > > arch/x86/include/asm/extable_fixup_types.h | 4 +- > > arch/x86/include/asm/fred.h | 131 ++++++++++ > > arch/x86/include/asm/idtentry.h | 76 +++++- > > arch/x86/include/asm/irq.h | 5 + > > arch/x86/include/asm/irq_vectors.h | 15 +- > > arch/x86/include/asm/msr-index.h | 13 +- > > arch/x86/include/asm/processor.h | 12 +- > > arch/x86/include/asm/ptrace.h | 36 ++- > > arch/x86/include/asm/switch_to.h | 10 +- > > arch/x86/include/asm/thread_info.h | 35 +-- > > arch/x86/include/asm/traps.h | 13 + > > arch/x86/include/asm/vmx.h | 17 +- > > arch/x86/include/uapi/asm/processor-flags.h | 2 + > > arch/x86/kernel/Makefile | 1 + > > arch/x86/kernel/apic/apic.c | 11 +- > > arch/x86/kernel/apic/vector.c | 8 +- > > arch/x86/kernel/cpu/acrn.c | 7 +- > > arch/x86/kernel/cpu/common.c | 88 ++++--- > > arch/x86/kernel/cpu/mce/core.c | 11 + > > arch/x86/kernel/cpu/mshyperv.c | 22 +- > > arch/x86/kernel/espfix_64.c | 8 + > > arch/x86/kernel/fred.c | 73 ++++++ > > arch/x86/kernel/head_32.S | 3 +- > > arch/x86/kernel/idt.c | 6 +- > > arch/x86/kernel/irq.c | 6 +- > > arch/x86/kernel/irqinit.c | 7 +- > > arch/x86/kernel/kvm.c | 4 +- > > arch/x86/kernel/nmi.c | 28 +++ > > arch/x86/kernel/process.c | 5 + > > arch/x86/kernel/process_64.c | 21 +- > > arch/x86/kernel/signal_32.c | 21 +- > > arch/x86/kernel/traps.c | 175 +++++++++++-- > > arch/x86/kvm/vmx/vmx.c | 33 ++- > > arch/x86/lib/x86-opcode-map.txt | 2 +- > > arch/x86/mm/extable.c | 28 +++ > > arch/x86/mm/fault.c | 20 +- > > drivers/xen/events/events_base.c | 5 +- > > kernel/fork.c | 6 + > > tools/arch/x86/include/asm/cpufeatures.h | 1 + > > .../arch/x86/include/asm/disabled-features.h | 8 +- > > tools/arch/x86/include/asm/msr-index.h | 13 +- > > tools/arch/x86/lib/x86-opcode-map.txt | 2 +- > > tools/objtool/arch/x86/decode.c | 19 +- > > 55 files changed, 1185 insertions(+), 175 deletions(-) > > create mode 100644 arch/x86/entry/entry_64_fred.S > > create mode 100644 arch/x86/entry/entry_fred.c > > create mode 100644 arch/x86/include/asm/event-type.h > > create mode 100644 arch/x86/include/asm/fred.h > > create mode 100644 arch/x86/kernel/fred.c > >
