On 05/03/2017 03:45 AM, Marc Zyngier wrote:
[Apologies for posting this at the beginning of a merge window, but as
this is a rather hot topic, I'd rather put it out as soon as possible]
Some systems have less than perfect GICv3 implementations, leading to
all kind of ugly issues (guest hanging, host dying). In order to allow
some level of diagnostic, and in some cases implement workarounds,
this series enables the trapping of both Group-0, Group-1 and Common
sysregs. Mediating the access at EL2 allows some form of sanity
checking that the HW is sometimes sorely lacking.
Instead of fully emulating a GICv3 CPU interface, we still use the
existing HW (list registers, AP registers, VMCR...), which allows the
code to be independent from the rest of the KVM code, and to cope with
partial trapping.
Of course, trapping has a cost, which is why this must be either
enabled on the command line, or selected by another cpu capability
(see Cavium erratum 30115). A quick test on an A57-based platform
shows a 25% hit when repeatedly banging on the trapped registers,
while normal workloads do not seem to suffer noticeably from such
trapping (hackbench variance is in the usual noise, despite being very
IPI happy).
This has been tested on a dual socket Thundex-X and a Freescale LS-2085a.
The first 6 patches are fixes, and only here for reference as they
have already been posted separately. The rest of the patches implement
Group-1, Group-0 and Common sysreg handlers, with the corresponding
command line options. I've also taken the liberty to rebase David
Daney's initial Cavium erratum 30115 workaround on top of this series,
and included it here as a typical use case.
Thanks Marc for working on this.
I tested this series based on your git branch:
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/gicv3-cpuif-mediated-access
This does indeed fix the problems we were seeing, so feel free to add:
Tested-by: David Daney <david.daney@xxxxxxxxxx>
to the entire series.
I would note that with these patches we see an occasional:
[ 3868.311491] Unexpected interrupt 4 on vcpu ffff801f57a6a020
[ 4262.063419] Unexpected interrupt 4 on vcpu ffff801f5b3dc040
[ 4262.063422] Unexpected interrupt 4 on vcpu ffff801f50972020
This is better than locking up the system, but I wonder if it indicates
that improvement to the code is still possible.
David Daney
David Daney (2):
arm64: Add MIDR values for Cavium cn83XX SoCs
arm64: Add workaround for Cavium Thunder erratum 30115
Marc Zyngier (29):
arm64: KVM: Fix decoding of Rt/Rt2 when trapping AArch32 CP accesses
arm64: KVM: Do not use stack-protector to compile EL2 code
arm: KVM: Do not use stack-protector to compile HYP code
KVM: arm/arm64: vgic-v2: Do not use Active+Pending state for a HW
interrupt
KVM: arm/arm64: vgic-v3: Do not use Active+Pending state for a HW
interrupt
KVM: arm/arm64: vgic-v3: Use PREbits to infer the number of
ICH_APxRn_EL2 registers
KVM: arm/arm64: vgic-v3: Add accessors for the ICH_APxRn_EL2 registers
arm64: Add a facility to turn an ESR syndrome into a sysreg encoding
KVM: arm64: Make kvm_condition_valid32() accessible from EL2
KVM: arm64: vgic-v3: Add hook to handle guest GICv3 sysreg accesses at
EL2
KVM: arm64: vgic-v3: Add ICV_BPR1_EL1 handler
KVM: arm64: vgic-v3: Add ICV_IGRPEN1_EL1 handler
KVM: arm64: vgic-v3: Add ICV_IAR1_EL1 handler
KVM: arm64: vgic-v3: Add ICV_EOIR1_EL1 handler
KVM: arm64: vgic-v3: Add ICV_AP1Rn_EL1 handler
KVM: arm64: vgic-v3: Add ICV_HPPIR1_EL1 handler
KVM: arm64: vgic-v3: Enable trapping of Group-1 system registers
KVM: arm64: Enable GICv3 Group-1 sysreg trapping via command-line
KVM: arm64: vgic-v3: Add ICV_BPR0_EL1 handler
KVM: arm64: vgic-v3: Add ICV_IGNREN0_EL1 handler
KVM: arm64: vgic-v3: Add misc Group-0 handlers
KVM: arm64: vgic-v3: Enable trapping of Group-0 system registers
KVM: arm64: Enable GICv3 Group-0 sysreg trapping via command-line
KVM: arm64: vgic-v3: Add ICV_DIR_EL1 handler
KVM: arm64: vgic-v3: Add ICV_RPR_EL1 handler
KVM: arm64: vgic-v3: Add ICV_CTLR_EL1 handler
KVM: arm64: vgic-v3: Add ICV_PMR_EL1 handler
KVM: arm64: Enable GICv3 common sysreg trapping via command-line
KVM: arm64: vgic-v3: Log which GICv3 system registers are trapped
Documentation/arm64/silicon-errata.txt | 1 +
arch/arm/kvm/hyp/Makefile | 2 +
arch/arm64/Kconfig | 11 +
arch/arm64/include/asm/cpucaps.h | 3 +-
arch/arm64/include/asm/cputype.h | 2 +
arch/arm64/include/asm/esr.h | 25 ++
arch/arm64/include/asm/kvm_emulate.h | 6 +
arch/arm64/include/asm/kvm_hyp.h | 1 +
arch/arm64/include/asm/sysreg.h | 9 +
arch/arm64/kernel/cpu_errata.c | 21 +
arch/arm64/kvm/hyp/Makefile | 2 +
arch/arm64/kvm/hyp/switch.c | 14 +
arch/arm64/kvm/sys_regs.c | 8 +-
include/kvm/arm_vgic.h | 1 +
include/linux/irqchip/arm-gic-v3.h | 6 +
virt/kvm/arm/aarch32.c | 2 +-
virt/kvm/arm/hyp/vgic-v3-sr.c | 792 +++++++++++++++++++++++++++++++--
virt/kvm/arm/vgic/vgic-v2.c | 7 +
virt/kvm/arm/vgic/vgic-v3.c | 52 +++
19 files changed, 934 insertions(+), 31 deletions(-)
