Virtual machines under Xen Hypervisor (DomU) running in Xen PV mode use a
special, nonstandard synthetized CPU topology which "just works" under
kernels 6.9.x while newer kernels wrongly assuming a "crash kernel" and
disable SMP (reducing to one CPU core) because the newer topology
implementation produces a wrong error "[Firmware Bug]: APIC enumeration
order not specification compliant" after new topology checks which are
improper for Xen PV platform. As a result, the kernel disables SMP and
activates just one CPU core within the PV DomU "VM" (DomU in PV mode).
The patch disables the regarding checks if it is running in Xen PV
mode (only) and bring back SMP / all CPUs as in the past to such DomU
VMs. The Xen subsystem takes care of the proper interaction between "guest"
(DomU) and the "host" (Dom0).
Signed-off-by: Niels Dettenbach <nd@xxxxxxxxxxxx>
---
The current behaviour leads all of our production Xen Host platforms
(amd64 - HPE proliant) unusable after updating to newer linux kernels
(with just one CPU available/activated per VM) while older kernels and
other OS (current NetBSD PV DomU) still work fully (and stable since many
years on the platform).
Xen PV mode is still provided by current Xen and widely used - even
if less wide as the newer Xen PVH mode today. So a solution probably
will be required and the other exemptions in topology.c seems to support that
as well.
So we assume that bug affects stable@xxxxxxxxxxxxxxx as well.
dmesg from affected kernel on affected DomU PV:
-- snip --
[ 0.640364] CPU topo: Enumerated BSP APIC 0 is not marked in APICBASE MSR
[ 0.640367] CPU topo: Assuming crash kernel. Limiting to one CPU to
prevent machine INIT
[ 0.640368] CPU topo: [Firmware Bug]: APIC enumeration order not
specification compliant
[ 0.640376] CPU topo: Max. logical packages: 1
[ 0.640378] CPU topo: Max. logical dies: 1
[ 0.640379] CPU topo: Max. dies per package: 1
[ 0.640386] CPU topo: Max. threads per core: 1
[ 0.640388] CPU topo: Num. cores per package: 1
[ 0.640389] CPU topo: Num. threads per package: 1
[ 0.640390] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs
[ 0.640402] Cannot find an available gap in the 32-bit address range
-- snap --
after patch applied:
-- snip --
[ 0.369439] CPU topo: Max. logical packages: 1
[ 0.369441] CPU topo: Max. logical dies: 1
[ 0.369442] CPU topo: Max. dies per package: 1
[ 0.369450] CPU topo: Max. threads per core: 2
[ 0.369452] CPU topo: Num. cores per package: 3
[ 0.369453] CPU topo: Num. threads per package: 6
[ 0.369453] CPU topo: Allowing 6 present CPUs plus 0 hotplug CPUs
-- snap --
We tested the patch intensely under productive / high load since 2 weeks now
with no issues (no crashes emulated).
references:
arch/x86/kernel/cpu/topology.c
[line 448]
-- snip --
/*
* XEN PV is special as it does not advertise the local APIC
* properly, but provides a fake topology for it so that the
* infrastructure works. So don't apply the restrictions vs. APIC
* here.
*/
--snap --
Am Dienstag, 1. Oktober 2024, 14:28:58 schrieben Sie:
> Please Cc LKML on such things as docuemnted...
>
> On Mon, Sep 30 2024 at 09:43, Niels Dettenbach wrote:
> > Virtual machines under Xen Hypervisor (DomU) running in Xen PV mode use a
> > special, nonstandard synthetized CPU topology which "just works" under
> > kernels 6.9.x while newer kernels wrongly assuming a "crash kernel" and
> > disable SMP (reducing to one CPU core) because the newer topology
> > implementation produces a wrong error "[Firmware Bug]: APIC enumeration
> > order not specification compliant" after new topology checks which are
> > improper for Xen PV platform.
>
> Why are they incorrect for XENPV? If the hypervisor exposes a topology
> via CPUID then that topology wants to be correct. Everything else is a
> firmware bug. And no, we are not papering over that.
>
> > As a result, the kernel disables SMT and activates just one CPU core
> > within the VM (DomU).
>
> I'm not seing how that happens due to the firmware bug issue. That's a
> different problem and has nothing to do with
> topology_register[_boot]_apic().
>
> > The patch disables the regarding checks if it is running in Xen PV
> > mode (only) and bring back SMP / all CPUs as in the past to such DomU
> > VMs.
>
> So what enumerates the APICs on your systems? Is the topology exposed
> via ACPI? If not, then this can't happen at all.
>
> And please report this with LKML cc'ed with the above questions
> answered.
Hi Thomas,
my patch just extends existing exceptions for xen in topology.c
i took the exact inline explanation of my patch from existing (so i assume
approved) code in topology.c - please have a look (line 449 of arch/x86/
kernel/cpu/topology.c).
xen is "part" of the linux kernel and in PV mode DomU (guest) the xen Dom0
(„host“) apic topology is "emulated" by Xen itself in a „non-standard“ but
working structure while Xen in PV DomU takes care of the frontend side. The
patch only affects this very special constellation / setup.
In production environments different kernel / xen versions typically are used
- means: even if the Xen project will „fix“ / or adapt to that new behaviour
of linux kernel, it still breaks compatibility on affected existing Xen (PV)
productive installations for at least several years.
The >6.9 kernel runs SMP fine if i boot it on the same bare metal of that
different HP proliant machines (means instead of booting it as Xen PV guest
onto a Linux xen Dom0 on that hardware) so the hardwares firmware could not
be the issue here, while it is affected in Xen DomU VM guests (in PV mode)
only on different hardware with different (but older) Xen versions / binaries
(in Xen PV mode only). So from my prospective and as far as i read the whole
topology.c (i'm not a skilled kernel developer) which helds several such
exemptions from different tests/checks / fnctions this is a (known) Xen
specialty.
I know that Xen PV mode gets more succeeded by the newer xen PVHM mode these
days, but PV mode is officially supported by Xen as linux kernel and still
used in productive scenarios.
see i.e.:
https://xen-orchestra.com/blog/xen-virtualization-modes/
https://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum
many thanks for your time,
niels.
--- linux/arch/x86/kernel/cpu/topology.c.orig 2024-09-11 09:53:16.194095250
+0200
+++ linux/arch/x86/kernel/cpu/topology.c 2024-09-30 09:39:11.041326786
+0200
@@ -131,6 +131,18 @@ static __init bool check_for_real_bsp(u3
bool is_bsp = false, has_apic_base = boot_cpu_data.x86 >= 6;
u64 msr;
+
+ /*
+ * XEN PV is special as it does not advertise the local APIC
+ * properly, but provides a fake topology for it so that the
+ * infrastructure works. So don't apply the restrictions vs. APIC
+ * here.
+ */
+ if (xen_pv_domain()) {
+ topo_info.real_bsp_apic_id = topo_info.boot_cpu_apic_id;
+ return false;
+ }
+
/*
* There is no real good way to detect whether this a kdump()
* kernel, but except on the Voyager SMP monstrosity which is not
