Hi!
This is a complete rework of the parallel bringup patch series (V17)
https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@xxxxxxxxxxxxx
to address the issues which were discovered in review:
1) The X86 microcode loader serialization requirement
https://lore.kernel.org/lkml/87v8iirxun.ffs@tglx
Microcode loading on HT enabled X86 CPUs requires that the microcode is
loaded on the primary thread. The sibling thread(s) must be in
quiescent state; either looping in a place which is aware of potential
changes by the microcode update (see late loading) or in fully quiescent
state, i.e. waiting for INIT/SIPI.
This is required by hardware/firmware on Intel. Aside of that it's a
vendor independent software correctness issue. Assume the following
sequence:
CPU1.0 CPU1.1
CPUID($A)
Load microcode.
Changes CPUID($A, $B)
CPUID($B)
CPU1.1 makes a decision on $A and $B which might be inconsistent due
to the microcode update.
The solution for this is to bringup the primary threads first and after
that the siblings. Loading microcode on the siblings is a NOOP on Intel
and on AMD it is guaranteed to only modify thread local state.
This ensures that the APs can load microcode before reaching the alive
synchronization point w/o doing any further x86 specific
synchronization between the core siblings.
2) The general design issues discussed in V16
https://lore.kernel.org/lkml/87pm8y6yme.ffs@tglx
The previous parallel bringup patches just glued this mechanism into
the existing code without a deeper analysis of the synchronization
mechanisms and without generalizing it so that the control logic is
mostly in the core code and not made an architecture specific tinker
space.
Much of that had been pointed out 2 years ago in the discussions about
the early versions of parallel bringup already.
The series is based on:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip x86/apic
and also available from git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git hotplug
Background
----------
The reason why people are interested in parallel bringup is to shorten
the (kexec) reboot time of cloud servers to reduce the downtime of the
VM tenants. There are obviously other interesting use cases for this
like VM startup time, embedded devices...
The current fully serialized bringup does the following per AP:
1) Prepare callbacks (allocate, intialize, create threads)
2) Kick the AP alive (e.g. INIT/SIPI on x86)
3) Wait for the AP to report alive state
4) Let the AP continue through the atomic bringup
5) Let the AP run the threaded bringup to full online state
There are two significant delays:
#3 The time for an AP to report alive state in start_secondary() on x86
has been measured in the range between 350us and 3.5ms depending on
vendor and CPU type, BIOS microcode size etc.
#4 The atomic bringup does the microcode update. This has been measured
to take up to ~8ms on the primary threads depending on the microcode
patch size to apply.
On a two socket SKL server with 56 cores (112 threads) the boot CPU spends
on current mainline about 800ms busy waiting for the APs to come up and
apply microcode. That's more than 80% of the actual onlining procedure.
By splitting the actual bringup mechanism into two parts this can be
reduced to waiting for the first AP to report alive or if the system is
large enough the first AP is already waiting when the boot CPU finished the
wake-up of the last AP.
The actual solution comes in several parts
------------------------------------------
1) [P 1-2] General cleanups (init annotations, kernel doc...)
2) [P 3] The obvious
Avoid pointless delay calibration when TSC is synchronized across
sockets. That removes a whopping 100ms delay for the first CPU of a
socket. This is an improvement independent of parallel bringup and had
been discussed two years ago already.
2) [P 3-6] Removal of the CPU0 hotplug hack.
This was added 11 years ago with the promise to make this a real
hardware mechanism, but that never materialized. As physical CPU
hotplug is not really supported and the physical unplugging of CPU0
never materialized there is no reason to keep this cruft around. It's
just maintenance ballast for no value and the removal makes
implementing the parallel bringup feature way simpler.
3) [P 7-16] Cleanup of the existing bringup mechanism:
a) Code reorganisation so that the general hotplug specific code is
in smpboot.c and not sprinkled all over the place
b) Decouple MTRR/PAT initialization from smp_callout_mask to prepare
for replacing that mask with a hotplug core code synchronization
mechanism.
c) Make TSC synchronization function call based so that the control CPU
does not have to busy wait for nothing if synchronization is not
required.
d) Remove the smp_callin_mask synchronization point as its not longer
required due to #3c.
e) Rework the sparse_irq_lock held region in the core code so that the
next polling synchronization point in the x86 code can be removed to.
f) Due to #3e it's not longer required to spin wait for the AP to set
it's online bit. Remove wait_cpu_online() and the XENPV
counterpart. So the control CPU can directly wait for the online
idle completion by the AP and free the control CPU up for other
work.
This reduces the synchronization points in the x86 code to one, which
is the AP alive one. This synchronization will be moved to core
infrastructure in the next section.
4) [P 17-27] Replace the disconnected CPU state tracking
The extra CPU state tracking which is used by a few architectures is
completely separate from the CPU hotplug core code.
Replacing it by a variant integrated in the core hotplug machinery
allows to reduce architecture specific code and provides a generic
synchronization mechanism for (parallel) CPU bringup/teardown.
- Convert x86 over and replace the AP alive synchronization on x86 with
the core variant which removes the remaining x86 hotplug
synchronization masks.
- Convert the other architectures usage and remove the old interface
and code.
5) [P 28-30] Split the bringup into two steps
First step invokes the wakeup function on the BP, e.g. SIPI/STARTUP on
x86. The second one waits on the BP for the AP to report alive and
releases it for the complete onlining.
As the hotplug state machine allows partial bringup this allows later
to kick all APs alive in a first iteration and then bring them up
completely one by one afterwards.
6) [P 31] Switch the primary thread detection to a cpumask
This makes the parallel bringup a simple cpumask based mechanism
without tons of conditionals and checks for primary threads.
7) [P 32] Implement the parallel bringup core code
The parallel bringup looks like this:
1) Bring up the primary SMT threads to the CPUHP_KICK_AP_ALIVE step
one by one
2) Bring up the primary SMT threads to the CPUHP_ONLINE step one by
one
3) Bring up the secondary SMT threads to the CPUHP_KICK_AP_ALIVE
step one by one
4) Bring up the secondary SMT threads to the CPUHP_ONLINE
step one by one
In case that SMT is not supported this is obviously reduced to step #1
and #2.
8) [P 33-37] Prepare X86 for parallel bringup and enable it
Caveats
-------
The non X86 changes have been all compile tested. Boot and runtime
testing has only be done on a few real hardware platforms and qemu as
available. That definitely needs some help from the people who have
these systems at their fingertips.
Results and analysis
--------------------
Here are numbers for a dual socket SKL 56 cores/ 112 threads machine. All
numbers in milliseconds. The time measured is the time which the cpu_up()
call takes for each CPU and phase. It's not exact as the system is already
scheduling, handling interrupts and soft interrupts, which is obviously
skewing the picture slightly.
Baseline tip tree x86/apic branch.
total avg/CPU min max
total : 912.081 8.217 3.720 113.271
The max of 100ms is due to the silly delay calibration for the second
socket which takes 100ms and was eliminated first. Also the other initial
cleanups and improvements take some time away.
So the real baseline becomes:
total avg/CPU min max
total : 785.960 7.081 3.752 36.098
The max here is on the first CPU of the second socket. 20ms of that is due
to TSC synchronization and an extra 2ms to react on the SIPI.
With parallel bootup enabled this becomes:
total avg/CPU min max
prepare: 39.108 0.352 0.238 0.883
online : 45.166 0.407 0.170 20.357
total : 84.274 0.759 0.408 21.240
That's a factor ~9.3 reduction on average.
Looking at the 27 primary threads of socket 0 then this becomes even more
interesting:
total avg/CPU min max
total : 325.764 12.065 11.981 14.125
versus:
total avg/CPU min max
prepare: 8.945 0.331 0.238 0.834
online : 4.830 0.179 0.170 0.212
total : 13.775 0.510 0.408 1.046
So the reduction factor is ~23.5 here. That's mostly because the 20ms TSC
sync is not skewing the picture.
For all 55 primaries, i.e with the 20ms TSC sync extra for socket 1 this
becomes:
total avg/CPU min max
total : 685.489 12.463 11.975 36.098
versus:
total avg/CPU min max
prepare: 19.080 0.353 0.238 0.883
online : 30.283 0.561 0.170 20.357
total : 49.363 0.914 0.408 21.240
The TSC sync reduces the win to a factor of ~13.8
With 'tsc=reliable' on the command line the socket sync is disabled which
brings it back to the socket 0 numbers:
total avg/CPU min max
prepare: 18.970 0.351 0.231 0.874
online : 10.328 0.191 0.169 0.358
total : 29.298 0.543 0.400 1.232
Now looking at the secondary threads only:
total avg/CPU min max
total : 100.471 1.794 0.375 4.745
versus:
total avg/CPU min max
prepare: 19.753 0.353 0.257 0.512
online : 14.671 0.262 0.179 3.461
total : 34.424 0.615 0.436 3.973
Still a factor of ~3.
The average on the secondaries for the serialized bringup is significantly
lower than for the primaries because the SIPI response time is shorter and
the microcode update takes no time.
This varies wildly with the system, whether microcode in BIOS is already up
to date, how big the microcode patch is and how long the INIT/SIPI response
time is. On an AMD Zen3 machine INIT/SIPI response time is amazingly fast
(350us), but then it lacks TSC_ADJUST and does a two millisecond TSC sync
test for _every_ AP. All of this sucks...
Possible further enhancements
-----------------------------
It's definitely worthwhile to look into reducing the cross socket TSC sync
test time. It's probably safe enough to use 5ms or even 2ms instead of 20ms
on systems with TSC_ADJUST and a few other 'TSC is sane' indicators. Moving
it out of the hotplug path is eventually possible, but that needs some deep
thoughts.
Let's take the TSC sync out of the picture by adding 'tsc=reliable" to the
kernel command line. So the bringup of 111 APs takes:
total avg/CPU min max
prepare: 38.936 0.351 0.231 0.874
online : 25.231 0.227 0.169 3.465
total : 64.167 0.578 0.400 4.339
Some of the outliers are not necessarily in the state callbacks as the
system is already scheduling and handles interrupts and soft
interrupts. Haven't analyzed that yet in detail.
In the prepare stage which runs on the control CPU the larger steps are:
smpcfd:prepare 16us avg/CPU
threads:prepare 98us avg/CPU
workqueue:prepare 43us avg/CPU
trace/RB:prepare 135us avg/CPU
The trace ringbuffer initialization allocates 354 pages and 354 control
structures one by one. That probably should allocate a large page and an
array of control structures and work from there. I'm sure that would reduce
this significantly. Steven?
smpcfd does just a percpu allocation. No idea why that takes that long.
Vs. threads and workqueues. David thought about spreading out the
preparation work and do it really in parallel. That's a nice idea, but the
threads and workqueue prepare steps are self serializing. The workqueue one
has a global mutex and aside of that both steps create kernel threads which
implicitely serialize on kthreadd. alloc_percpu(), which is used by
smpcfd:prepare is also globally serialized.
The rest of the prepare steps is pretty much in the single digit
microseconds range.
On the AP side it should be possible to move some of the initialization
steps before the alive synchronization point, but that really needs a lot
of analysis whether the functions are safe to invoke that early and outside
of the cpu_hotplug_lock held region for the case of two stage parallel
bringup; see below.
The largest part is:
identify_secondary_cpu() 99us avg/CPU
Inside of identify_secondary_cpu() the largest offender:
mcheck_init() 73us avg/CPU
This part is definitly worth to be looked at whether it can be at least
partially moved to the early startup code before the alive
synchronization point. There's a lot of deep analysis required and
ideally we just rewrite the whole CPUID evaluation trainwreck
completely.
The rest of the AP side is low single digit microseconds except of:
perf/x86:starting 14us avg/CPU
smpboot/threads:online 13us avg/CPU
workqueue:online 17us avg/CPU
mm/vmstat:online 17us avg/CPU
sched:active 30us avg/CPU
sched:active is special. Onlining the first secondary HT thread on the
second socket creates a 3.2ms outlier which skews the whole picture. That's
caused by enabling the static key sched_smt_present which patches the world
and some more. For all other APs this is really in the 1us range. This
definitely could be postponed during bootup like the scheduler domain
rebuild is done after the bringup. But that's still fully serialized and
single threaded and obviously could be done later in the context of async
parallel init. It's unclear why this is different with the fully serialized
bringup where it takes significantly less time, but that's something which
needs to be investigated.
Is truly parallel bringup feasible?
-----------------------------------
In theory yes, realistically no. Why?
1) The preparation phase
Allocating memory, creating threads for the to be brought up CPU must
obviously happen on an already online CPU.
While it would be possible to bring up a subset of CPUs first and let
them do the preparation steps for groups of still offline CPUs
concurrently, the actual benefit of doing so is dubious.
The prime example is kernel thread creation, which is implicitely
serialized on kthreadd.
A simple experiment shows that 4 concurrent workers on 4 different
CPUs where each is creating 14 * 5 = 70 kernel threads are 5% slower
than a single worker creating 4 * 14 * 5 = 280 threads.
So we'd need to have multiple kthreadd instances to handle that,
which would then serialize on tasklist lock and other things.
That aside the preparation phase is also affected by the problem
below.
2) Assumptions about hotplug serialization
a) There are quite some assumptions about CPU bringup being fully
serialized across state transitions. A lot of state callbacks rely
on that and would require local locking.
Adding that local locking is surely possible, but that has several
downsides:
- It adds complexity and makes it harder for developers to get
this correct. The subtle bugs resulting out of that are going
to be interesting
- Fine grained locking has a charm, but only if the time spent
for the actual work is larger than the time required for
serialization and synchronization.
Serializing a callback which takes less than a microsecond and
then having a large number of CPUs contending on the lock will
not make it any faster at all. That's a well known issue of
parallelizing and neither made up nor kernel specific.
b) Some operations definitely require to be protected by the
cpu_hotplug_lock, especially those which affect cpumasks as the
masks are guaranteed to be stable in a cpus_read_lock()'ed region.
As this lock cannot be taken in atomic contexts, it's required
that the control CPU holds the lock write locked across these
state transitions. And no, we are not making this a spinlock just
for that and we even can't.
Just slapping a lock into the x86 specific part of the cpumask
update function does not solve anything. The relevant patch in V17
is completely useless as it only serializes the actual cpumask/map
modifications, but all read side users are hosed if the update
would be moved before the alive synchronization point, i.e. into a
non hotplug lock protected region.
Even if the hotplug lock would be held accross the whole parallel
bringup operation then this would still expose all usage of these
masks and maps in the actual hotplug state callbacks to concurrent
modifications.
And no, we are not going to expose an architecture specific raw
spinlock to the hotplug state callbacks, especially not to those
in generic code.
c) Some cpu_read_lock()'ed regions also expect that there is no CPU
state transition happening which would modify their local
state. This would again require local serialization.
3) The amount of work and churn:
- Analyze the per architecture low level startup functions plus their
descendant functions and make them ready for concurrency if
necessary.
- Analyze ~300 hotplug state callbacks and their descendant functions
and make them ready for concurrency if necessary.
- Analyze all cpus_read_lock()'ed regions and address their
requirements.
- Rewrite the core code to handle the cpu_hotplug_lock requirements
only in distinct phases of the state machine.
- Rewrite the core code to handle state callback failure and the
related rollback in the context of the new rules.
- ...
Even if some people are dedicated enough to do that, it's very
questionable whether the resulting complexity is justified.
We've spent a serious amount of time to sanitize hotplug and bring it
into a state where it is correct. This also made it reasonably simple
for developers to implement hotplug state callbacks without having to
become hotplug experts.
Breaking this completely up will result in a flood of hard to diagnose
subtle issues for sure. Who is going to deal with them?
The experience with this series so far does not make me comfortable
about that thought in any way.
Summary
-------
The obvious and low hanging fruits have to be solved first:
- The CPUID evaluation and related setup mechanisms
- The trace/ringbuffer oddity
- The sched:active oddity for the first sibling on the second socket
- Some other expensive things which I'm not seeing in my test setup due
to lack of hardware or configuration.
Anything else is pretty much wishful thinking in my opinion.
To be clear. I'm not standing in the way if there is a proper solution,
but that requires to respect the basic engineering rules:
1) Correctness first
2) Keep it maintainable
3) Keep it simple
So far this stuff failed already at #1.
I completely understand why this is important for cloud people, but
the real question to ask here is what are the actual requirements.
As far as I understand the main goal is to make a (kexec) reboot
almost invisible to VM tenants.
Now lets look at how this works:
A) Freeze VMs and persist state
B) kexec into the new kernel
C) Restore VMs from persistant memory
D) Thaw VMs
So the key problem is how long it takes to get from #B to #C and finally
to #D.
As far as I understand #C takes a serious amount of time and cannot be
parallelized for whatever reasons.
At the same time the number of online CPUs required to restore the VMs
state is less than the number of online CPUs required to actually
operate them in #D.
That means it would be good enough to return to userspace with a
limited number of online CPUs as fast as possible. A certain amount of
CPUs are going to be busy with restoring the VMs state, i.e. one CPU
per VM. Some remaining non-busy CPU can bringup the rest of the system
and the APs in order to be functional for #D, i.e the restore of VM
operation.
Trying to optimize this purely in kernel space by adding complexity of
dubious value is simply bogus in my opinion.
It's already possible today to limit the number of CPUs which are
initially onlined and online the rest later from user space.
There are two issue there:
a) The death by MCE broadcast problem
Quite some (contemporary) x86 CPU generations are affected by
this:
- MCE can be broadcasted to all CPUs and not only issued locally
to the CPU which triggered it.
- Any CPU which has CR4.MCE == 0, even if it sits in a wait
for INIT/SIPI state, will cause an immediate shutdown of the
machine if a broadcasted MCE is delivered.
b) Do the parallel bringup via sysfs control knob
The per CPU target state interface allows to do that today one
by one, but it's akward and has quite some overhead.
A knob to online the rest of the not yet onlined present CPUs
with the benefit of the parallel bringup mechanism is
missing.
#a) That's a risk to take by the operator.
Even the regular serialized bringup does not protect against this
issue up to the point where all present CPUs have at least
initialized CR4.
Limiting the number of APs to online early via the kernel command
line widens that window and increases the risk further by
executing user space before all APs have CR4 initialized.
But the same applies to a deferred online mechanism implemented in
the kernel where some worker brings up the not yet online APs while
the early online CPUs are already executing user space code.
#b) Is a no brainer to implement on top of this.
Conclusion
----------
Adding the basic parallel bringup mechanism as provided by this series
makes a lot of sense. Improving particular issues as pointed out in the
analysis makes sense too.
But trying to solve an application specific problem fully in the kernel
with tons of complexity, without exploring straight forward and simple
approaches first, does not make any sense at all.
Thanks,
tglx
---
Documentation/admin-guide/kernel-parameters.txt | 20
Documentation/core-api/cpu_hotplug.rst | 13
arch/Kconfig | 23 +
arch/arm/Kconfig | 1
arch/arm/include/asm/smp.h | 2
arch/arm/kernel/smp.c | 18
arch/arm64/Kconfig | 1
arch/arm64/include/asm/smp.h | 2
arch/arm64/kernel/smp.c | 14
arch/csky/Kconfig | 1
arch/csky/include/asm/smp.h | 2
arch/csky/kernel/smp.c | 8
arch/mips/Kconfig | 1
arch/mips/cavium-octeon/smp.c | 1
arch/mips/include/asm/smp-ops.h | 1
arch/mips/kernel/smp-bmips.c | 1
arch/mips/kernel/smp-cps.c | 14
arch/mips/kernel/smp.c | 8
arch/mips/loongson64/smp.c | 1
arch/parisc/Kconfig | 1
arch/parisc/kernel/process.c | 4
arch/parisc/kernel/smp.c | 7
arch/riscv/Kconfig | 1
arch/riscv/include/asm/smp.h | 2
arch/riscv/kernel/cpu-hotplug.c | 14
arch/x86/Kconfig | 45 --
arch/x86/include/asm/apic.h | 5
arch/x86/include/asm/cpu.h | 5
arch/x86/include/asm/cpumask.h | 5
arch/x86/include/asm/processor.h | 1
arch/x86/include/asm/realmode.h | 3
arch/x86/include/asm/sev-common.h | 3
arch/x86/include/asm/smp.h | 26 -
arch/x86/include/asm/topology.h | 23 -
arch/x86/include/asm/tsc.h | 2
arch/x86/kernel/acpi/sleep.c | 9
arch/x86/kernel/apic/apic.c | 22 -
arch/x86/kernel/callthunks.c | 4
arch/x86/kernel/cpu/amd.c | 2
arch/x86/kernel/cpu/cacheinfo.c | 21
arch/x86/kernel/cpu/common.c | 50 --
arch/x86/kernel/cpu/topology.c | 3
arch/x86/kernel/head_32.S | 14
arch/x86/kernel/head_64.S | 121 +++++
arch/x86/kernel/sev.c | 2
arch/x86/kernel/smp.c | 3
arch/x86/kernel/smpboot.c | 508 ++++++++----------------
arch/x86/kernel/topology.c | 98 ----
arch/x86/kernel/tsc.c | 20
arch/x86/kernel/tsc_sync.c | 36 -
arch/x86/power/cpu.c | 37 -
arch/x86/realmode/init.c | 3
arch/x86/realmode/rm/trampoline_64.S | 27 +
arch/x86/xen/enlighten_hvm.c | 11
arch/x86/xen/smp_hvm.c | 16
arch/x86/xen/smp_pv.c | 56 +-
drivers/acpi/processor_idle.c | 4
include/linux/cpu.h | 4
include/linux/cpuhotplug.h | 17
kernel/cpu.c | 397 +++++++++++++++++-
kernel/smp.c | 2
kernel/smpboot.c | 163 -------
62 files changed, 953 insertions(+), 976 deletions(-)
