On 15.04.23 01:44, Thomas Gleixner wrote:
Hi! This is a complete rework of the parallel bringup patch series (V17) https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@xxxxxxxxxxxxx to address the issues which were discovered in review: 1) The X86 microcode loader serialization requirement https://lore.kernel.org/lkml/87v8iirxun.ffs@tglx Microcode loading on HT enabled X86 CPUs requires that the microcode is loaded on the primary thread. The sibling thread(s) must be in quiescent state; either looping in a place which is aware of potential changes by the microcode update (see late loading) or in fully quiescent state, i.e. waiting for INIT/SIPI. This is required by hardware/firmware on Intel. Aside of that it's a vendor independent software correctness issue. Assume the following sequence: CPU1.0 CPU1.1 CPUID($A) Load microcode. Changes CPUID($A, $B) CPUID($B) CPU1.1 makes a decision on $A and $B which might be inconsistent due to the microcode update. The solution for this is to bringup the primary threads first and after that the siblings. Loading microcode on the siblings is a NOOP on Intel and on AMD it is guaranteed to only modify thread local state. This ensures that the APs can load microcode before reaching the alive synchronization point w/o doing any further x86 specific synchronization between the core siblings. 2) The general design issues discussed in V16 https://lore.kernel.org/lkml/87pm8y6yme.ffs@tglx The previous parallel bringup patches just glued this mechanism into the existing code without a deeper analysis of the synchronization mechanisms and without generalizing it so that the control logic is mostly in the core code and not made an architecture specific tinker space. Much of that had been pointed out 2 years ago in the discussions about the early versions of parallel bringup already. The series is based on: git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip x86/apic and also available from git: git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git hotplug Background ---------- The reason why people are interested in parallel bringup is to shorten the (kexec) reboot time of cloud servers to reduce the downtime of the VM tenants. There are obviously other interesting use cases for this like VM startup time, embedded devices... The current fully serialized bringup does the following per AP: 1) Prepare callbacks (allocate, intialize, create threads) 2) Kick the AP alive (e.g. INIT/SIPI on x86) 3) Wait for the AP to report alive state 4) Let the AP continue through the atomic bringup 5) Let the AP run the threaded bringup to full online state There are two significant delays: #3 The time for an AP to report alive state in start_secondary() on x86 has been measured in the range between 350us and 3.5ms depending on vendor and CPU type, BIOS microcode size etc. #4 The atomic bringup does the microcode update. This has been measured to take up to ~8ms on the primary threads depending on the microcode patch size to apply. On a two socket SKL server with 56 cores (112 threads) the boot CPU spends on current mainline about 800ms busy waiting for the APs to come up and apply microcode. That's more than 80% of the actual onlining procedure. By splitting the actual bringup mechanism into two parts this can be reduced to waiting for the first AP to report alive or if the system is large enough the first AP is already waiting when the boot CPU finished the wake-up of the last AP. The actual solution comes in several parts ------------------------------------------ 1) [P 1-2] General cleanups (init annotations, kernel doc...) 2) [P 3] The obvious Avoid pointless delay calibration when TSC is synchronized across sockets. That removes a whopping 100ms delay for the first CPU of a socket. This is an improvement independent of parallel bringup and had been discussed two years ago already. 2) [P 3-6] Removal of the CPU0 hotplug hack. This was added 11 years ago with the promise to make this a real hardware mechanism, but that never materialized. As physical CPU hotplug is not really supported and the physical unplugging of CPU0 never materialized there is no reason to keep this cruft around. It's just maintenance ballast for no value and the removal makes implementing the parallel bringup feature way simpler. 3) [P 7-16] Cleanup of the existing bringup mechanism: a) Code reorganisation so that the general hotplug specific code is in smpboot.c and not sprinkled all over the place b) Decouple MTRR/PAT initialization from smp_callout_mask to prepare for replacing that mask with a hotplug core code synchronization mechanism. c) Make TSC synchronization function call based so that the control CPU does not have to busy wait for nothing if synchronization is not required. d) Remove the smp_callin_mask synchronization point as its not longer required due to #3c. e) Rework the sparse_irq_lock held region in the core code so that the next polling synchronization point in the x86 code can be removed to. f) Due to #3e it's not longer required to spin wait for the AP to set it's online bit. Remove wait_cpu_online() and the XENPV counterpart. So the control CPU can directly wait for the online idle completion by the AP and free the control CPU up for other work. This reduces the synchronization points in the x86 code to one, which is the AP alive one. This synchronization will be moved to core infrastructure in the next section. 4) [P 17-27] Replace the disconnected CPU state tracking The extra CPU state tracking which is used by a few architectures is completely separate from the CPU hotplug core code. Replacing it by a variant integrated in the core hotplug machinery allows to reduce architecture specific code and provides a generic synchronization mechanism for (parallel) CPU bringup/teardown. - Convert x86 over and replace the AP alive synchronization on x86 with the core variant which removes the remaining x86 hotplug synchronization masks. - Convert the other architectures usage and remove the old interface and code. 5) [P 28-30] Split the bringup into two steps First step invokes the wakeup function on the BP, e.g. SIPI/STARTUP on x86. The second one waits on the BP for the AP to report alive and releases it for the complete onlining. As the hotplug state machine allows partial bringup this allows later to kick all APs alive in a first iteration and then bring them up completely one by one afterwards. 6) [P 31] Switch the primary thread detection to a cpumask This makes the parallel bringup a simple cpumask based mechanism without tons of conditionals and checks for primary threads. 7) [P 32] Implement the parallel bringup core code The parallel bringup looks like this:1) Bring up the primary SMT threads to the CPUHP_KICK_AP_ALIVE stepone by one 2) Bring up the primary SMT threads to the CPUHP_ONLINE step one by one 3) Bring up the secondary SMT threads to the CPUHP_KICK_AP_ALIVE step one by one 4) Bring up the secondary SMT threads to the CPUHP_ONLINE step one by one In case that SMT is not supported this is obviously reduced to step #1 and #2. 8) [P 33-37] Prepare X86 for parallel bringup and enable it Caveats ------- The non X86 changes have been all compile tested. Boot and runtime testing has only be done on a few real hardware platforms and qemu as available. That definitely needs some help from the people who have these systems at their fingertips. Results and analysis -------------------- Here are numbers for a dual socket SKL 56 cores/ 112 threads machine. All numbers in milliseconds. The time measured is the time which the cpu_up() call takes for each CPU and phase. It's not exact as the system is already scheduling, handling interrupts and soft interrupts, which is obviously skewing the picture slightly. Baseline tip tree x86/apic branch. total avg/CPU min max total : 912.081 8.217 3.720 113.271 The max of 100ms is due to the silly delay calibration for the second socket which takes 100ms and was eliminated first. Also the other initial cleanups and improvements take some time away. So the real baseline becomes: total avg/CPU min max total : 785.960 7.081 3.752 36.098 The max here is on the first CPU of the second socket. 20ms of that is due to TSC synchronization and an extra 2ms to react on the SIPI. With parallel bootup enabled this becomes: total avg/CPU min max prepare: 39.108 0.352 0.238 0.883 online : 45.166 0.407 0.170 20.357 total : 84.274 0.759 0.408 21.240 That's a factor ~9.3 reduction on average. Looking at the 27 primary threads of socket 0 then this becomes even more interesting: total avg/CPU min max total : 325.764 12.065 11.981 14.125 versus: total avg/CPU min max prepare: 8.945 0.331 0.238 0.834 online : 4.830 0.179 0.170 0.212 total : 13.775 0.510 0.408 1.046 So the reduction factor is ~23.5 here. That's mostly because the 20ms TSC sync is not skewing the picture. For all 55 primaries, i.e with the 20ms TSC sync extra for socket 1 this becomes: total avg/CPU min max total : 685.489 12.463 11.975 36.098 versus: total avg/CPU min max prepare: 19.080 0.353 0.238 0.883 online : 30.283 0.561 0.170 20.357 total : 49.363 0.914 0.408 21.240 The TSC sync reduces the win to a factor of ~13.8 With 'tsc=reliable' on the command line the socket sync is disabled which brings it back to the socket 0 numbers: total avg/CPU min max prepare: 18.970 0.351 0.231 0.874 online : 10.328 0.191 0.169 0.358 total : 29.298 0.543 0.400 1.232 Now looking at the secondary threads only: total avg/CPU min max total : 100.471 1.794 0.375 4.745 versus: total avg/CPU min max prepare: 19.753 0.353 0.257 0.512 online : 14.671 0.262 0.179 3.461 total : 34.424 0.615 0.436 3.973 Still a factor of ~3. The average on the secondaries for the serialized bringup is significantly lower than for the primaries because the SIPI response time is shorter and the microcode update takes no time. This varies wildly with the system, whether microcode in BIOS is already up to date, how big the microcode patch is and how long the INIT/SIPI response time is. On an AMD Zen3 machine INIT/SIPI response time is amazingly fast (350us), but then it lacks TSC_ADJUST and does a two millisecond TSC sync test for _every_ AP. All of this sucks... Possible further enhancements ----------------------------- It's definitely worthwhile to look into reducing the cross socket TSC sync test time. It's probably safe enough to use 5ms or even 2ms instead of 20ms on systems with TSC_ADJUST and a few other 'TSC is sane' indicators. Moving it out of the hotplug path is eventually possible, but that needs some deep thoughts. Let's take the TSC sync out of the picture by adding 'tsc=reliable" to the kernel command line. So the bringup of 111 APs takes: total avg/CPU min max prepare: 38.936 0.351 0.231 0.874 online : 25.231 0.227 0.169 3.465 total : 64.167 0.578 0.400 4.339 Some of the outliers are not necessarily in the state callbacks as the system is already scheduling and handles interrupts and soft interrupts. Haven't analyzed that yet in detail. In the prepare stage which runs on the control CPU the larger steps are: smpcfd:prepare 16us avg/CPU threads:prepare 98us avg/CPU workqueue:prepare 43us avg/CPU trace/RB:prepare 135us avg/CPU The trace ringbuffer initialization allocates 354 pages and 354 control structures one by one. That probably should allocate a large page and an array of control structures and work from there. I'm sure that would reduce this significantly. Steven? smpcfd does just a percpu allocation. No idea why that takes that long. Vs. threads and workqueues. David thought about spreading out the preparation work and do it really in parallel. That's a nice idea, but the threads and workqueue prepare steps are self serializing. The workqueue one has a global mutex and aside of that both steps create kernel threads which implicitely serialize on kthreadd. alloc_percpu(), which is used by smpcfd:prepare is also globally serialized. The rest of the prepare steps is pretty much in the single digit microseconds range. On the AP side it should be possible to move some of the initialization steps before the alive synchronization point, but that really needs a lot of analysis whether the functions are safe to invoke that early and outside of the cpu_hotplug_lock held region for the case of two stage parallel bringup; see below. The largest part is: identify_secondary_cpu() 99us avg/CPUInside of identify_secondary_cpu() the largest offender:mcheck_init() 73us avg/CPU This part is definitly worth to be looked at whether it can be at least partially moved to the early startup code before the alive synchronization point. There's a lot of deep analysis required and ideally we just rewrite the whole CPUID evaluation trainwreck completely. The rest of the AP side is low single digit microseconds except of: perf/x86:starting 14us avg/CPU smpboot/threads:online 13us avg/CPU workqueue:online 17us avg/CPU mm/vmstat:online 17us avg/CPU sched:active 30us avg/CPU sched:active is special. Onlining the first secondary HT thread on the second socket creates a 3.2ms outlier which skews the whole picture. That's caused by enabling the static key sched_smt_present which patches the world and some more. For all other APs this is really in the 1us range. This definitely could be postponed during bootup like the scheduler domain rebuild is done after the bringup. But that's still fully serialized and single threaded and obviously could be done later in the context of async parallel init. It's unclear why this is different with the fully serialized bringup where it takes significantly less time, but that's something which needs to be investigated. Is truly parallel bringup feasible? ----------------------------------- In theory yes, realistically no. Why? 1) The preparation phase Allocating memory, creating threads for the to be brought up CPU must obviously happen on an already online CPU. While it would be possible to bring up a subset of CPUs first and let them do the preparation steps for groups of still offline CPUs concurrently, the actual benefit of doing so is dubious. The prime example is kernel thread creation, which is implicitely serialized on kthreadd. A simple experiment shows that 4 concurrent workers on 4 different CPUs where each is creating 14 * 5 = 70 kernel threads are 5% slower than a single worker creating 4 * 14 * 5 = 280 threads. So we'd need to have multiple kthreadd instances to handle that, which would then serialize on tasklist lock and other things. That aside the preparation phase is also affected by the problem below. 2) Assumptions about hotplug serialization a) There are quite some assumptions about CPU bringup being fully serialized across state transitions. A lot of state callbacks rely on that and would require local locking. Adding that local locking is surely possible, but that has several downsides: - It adds complexity and makes it harder for developers to get this correct. The subtle bugs resulting out of that are going to be interesting - Fine grained locking has a charm, but only if the time spent for the actual work is larger than the time required for serialization and synchronization. Serializing a callback which takes less than a microsecond and then having a large number of CPUs contending on the lock will not make it any faster at all. That's a well known issue of parallelizing and neither made up nor kernel specific. b) Some operations definitely require to be protected by the cpu_hotplug_lock, especially those which affect cpumasks as the masks are guaranteed to be stable in a cpus_read_lock()'ed region. As this lock cannot be taken in atomic contexts, it's required that the control CPU holds the lock write locked across these state transitions. And no, we are not making this a spinlock just for that and we even can't. Just slapping a lock into the x86 specific part of the cpumask update function does not solve anything. The relevant patch in V17 is completely useless as it only serializes the actual cpumask/map modifications, but all read side users are hosed if the update would be moved before the alive synchronization point, i.e. into a non hotplug lock protected region. Even if the hotplug lock would be held accross the whole parallel bringup operation then this would still expose all usage of these masks and maps in the actual hotplug state callbacks to concurrent modifications. And no, we are not going to expose an architecture specific raw spinlock to the hotplug state callbacks, especially not to those in generic code. c) Some cpu_read_lock()'ed regions also expect that there is no CPU state transition happening which would modify their local state. This would again require local serialization. 3) The amount of work and churn: - Analyze the per architecture low level startup functions plus their descendant functions and make them ready for concurrency if necessary. - Analyze ~300 hotplug state callbacks and their descendant functions and make them ready for concurrency if necessary. - Analyze all cpus_read_lock()'ed regions and address their requirements.- Rewrite the core code to handle the cpu_hotplug_lock requirementsonly in distinct phases of the state machine. - Rewrite the core code to handle state callback failure and the related rollback in the context of the new rules. - ... Even if some people are dedicated enough to do that, it's very questionable whether the resulting complexity is justified. We've spent a serious amount of time to sanitize hotplug and bring it into a state where it is correct. This also made it reasonably simple for developers to implement hotplug state callbacks without having to become hotplug experts. Breaking this completely up will result in a flood of hard to diagnose subtle issues for sure. Who is going to deal with them? The experience with this series so far does not make me comfortable about that thought in any way. Summary ------- The obvious and low hanging fruits have to be solved first: - The CPUID evaluation and related setup mechanisms - The trace/ringbuffer oddity - The sched:active oddity for the first sibling on the second socket- Some other expensive things which I'm not seeing in my test setup dueto lack of hardware or configuration. Anything else is pretty much wishful thinking in my opinion. To be clear. I'm not standing in the way if there is a proper solution, but that requires to respect the basic engineering rules: 1) Correctness first 2) Keep it maintainable 3) Keep it simple So far this stuff failed already at #1. I completely understand why this is important for cloud people, but the real question to ask here is what are the actual requirements. As far as I understand the main goal is to make a (kexec) reboot almost invisible to VM tenants. Now lets look at how this works: A) Freeze VMs and persist state B) kexec into the new kernel C) Restore VMs from persistant memory D) Thaw VMs So the key problem is how long it takes to get from #B to #C and finally to #D. As far as I understand #C takes a serious amount of time and cannot be parallelized for whatever reasons. At the same time the number of online CPUs required to restore the VMs state is less than the number of online CPUs required to actually operate them in #D. That means it would be good enough to return to userspace with a limited number of online CPUs as fast as possible. A certain amount of CPUs are going to be busy with restoring the VMs state, i.e. one CPU per VM. Some remaining non-busy CPU can bringup the rest of the system and the APs in order to be functional for #D, i.e the restore of VM operation. Trying to optimize this purely in kernel space by adding complexity of dubious value is simply bogus in my opinion. It's already possible today to limit the number of CPUs which are initially onlined and online the rest later from user space. There are two issue there: a) The death by MCE broadcast problem Quite some (contemporary) x86 CPU generations are affected by this: - MCE can be broadcasted to all CPUs and not only issued locally to the CPU which triggered it. - Any CPU which has CR4.MCE == 0, even if it sits in a wait for INIT/SIPI state, will cause an immediate shutdown of the machine if a broadcasted MCE is delivered. b) Do the parallel bringup via sysfs control knob The per CPU target state interface allows to do that today one by one, but it's akward and has quite some overhead. A knob to online the rest of the not yet onlined present CPUs with the benefit of the parallel bringup mechanism is missing. #a) That's a risk to take by the operator. Even the regular serialized bringup does not protect against this issue up to the point where all present CPUs have at least initialized CR4. Limiting the number of APs to online early via the kernel command line widens that window and increases the risk further by executing user space before all APs have CR4 initialized. But the same applies to a deferred online mechanism implemented in the kernel where some worker brings up the not yet online APs while the early online CPUs are already executing user space code. #b) Is a no brainer to implement on top of this. Conclusion ---------- Adding the basic parallel bringup mechanism as provided by this series makes a lot of sense. Improving particular issues as pointed out in the analysis makes sense too. But trying to solve an application specific problem fully in the kernel with tons of complexity, without exploring straight forward and simple approaches first, does not make any sense at all. Thanks, tglx --- Documentation/admin-guide/kernel-parameters.txt | 20 Documentation/core-api/cpu_hotplug.rst | 13 arch/Kconfig | 23 + arch/arm/Kconfig | 1 arch/arm/include/asm/smp.h | 2 arch/arm/kernel/smp.c | 18 arch/arm64/Kconfig | 1 arch/arm64/include/asm/smp.h | 2 arch/arm64/kernel/smp.c | 14 arch/csky/Kconfig | 1 arch/csky/include/asm/smp.h | 2 arch/csky/kernel/smp.c | 8 arch/mips/Kconfig | 1 arch/mips/cavium-octeon/smp.c | 1 arch/mips/include/asm/smp-ops.h | 1 arch/mips/kernel/smp-bmips.c | 1 arch/mips/kernel/smp-cps.c | 14 arch/mips/kernel/smp.c | 8 arch/mips/loongson64/smp.c | 1 arch/parisc/Kconfig | 1 arch/parisc/kernel/process.c | 4 arch/parisc/kernel/smp.c | 7 arch/riscv/Kconfig | 1 arch/riscv/include/asm/smp.h | 2 arch/riscv/kernel/cpu-hotplug.c | 14 arch/x86/Kconfig | 45 -- arch/x86/include/asm/apic.h | 5 arch/x86/include/asm/cpu.h | 5 arch/x86/include/asm/cpumask.h | 5 arch/x86/include/asm/processor.h | 1 arch/x86/include/asm/realmode.h | 3 arch/x86/include/asm/sev-common.h | 3 arch/x86/include/asm/smp.h | 26 - arch/x86/include/asm/topology.h | 23 - arch/x86/include/asm/tsc.h | 2 arch/x86/kernel/acpi/sleep.c | 9 arch/x86/kernel/apic/apic.c | 22 - arch/x86/kernel/callthunks.c | 4 arch/x86/kernel/cpu/amd.c | 2 arch/x86/kernel/cpu/cacheinfo.c | 21 arch/x86/kernel/cpu/common.c | 50 -- arch/x86/kernel/cpu/topology.c | 3 arch/x86/kernel/head_32.S | 14 arch/x86/kernel/head_64.S | 121 +++++ arch/x86/kernel/sev.c | 2 arch/x86/kernel/smp.c | 3 arch/x86/kernel/smpboot.c | 508 ++++++++---------------- arch/x86/kernel/topology.c | 98 ---- arch/x86/kernel/tsc.c | 20 arch/x86/kernel/tsc_sync.c | 36 - arch/x86/power/cpu.c | 37 - arch/x86/realmode/init.c | 3 arch/x86/realmode/rm/trampoline_64.S | 27 + arch/x86/xen/enlighten_hvm.c | 11 arch/x86/xen/smp_hvm.c | 16 arch/x86/xen/smp_pv.c | 56 +- drivers/acpi/processor_idle.c | 4 include/linux/cpu.h | 4 include/linux/cpuhotplug.h | 17 kernel/cpu.c | 397 +++++++++++++++++- kernel/smp.c | 2 kernel/smpboot.c | 163 ------- 62 files changed, 953 insertions(+), 976 deletions(-)
Tested with a Xen PV dom0 on an 8 cpu system, no issues found. Tested-by: Juergen Gross <jgross@xxxxxxxx> Juergen
Attachment:
OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key
Attachment:
OpenPGP_signature
Description: OpenPGP digital signature
