On 01/02/2022 20:53, David Woodhouse wrote:
Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for
them shaves about 80% off the AP bringup time on a 96-thread 2-socket
Skylake box (EC2 c5.metal) — from about 500ms to 100ms.
There are more wins to be had with further parallelisation, but this is
the simple part.
Hi,
We are interested in reducing the boot time of servers (with kexec), and
smpboot takes up a significant amount of time while booting. When
testing the patch series (rebased to v6.1) on a server with 128 CPUs
split across 2 NUMA nodes, it brought down the smpboot time from ~700ms
to 100ms. Adding another cpuhp state for do_wait_cpu_initialized to make
sure cpu_init is reached (as done in v1 of the series + using the
cpu_finishup_mask) brought it down further to ~30ms.
I just wanted to check what was needed to progress the patch series
further for review? There weren't any comments on v4 of the patch so I
couldn't figure out what more is needed. I think its quite useful to
have this working so would be really glad help in anything needed to
restart the review.
Thanks!
Usama
v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
in preparation for more parallelisation.
v4: Fixes to the real mode parallelisation patch spotted by SeanC, to
avoid scribbling on initial_gs in common_cpu_up(), and to allow all
24 bits of the physical X2APIC ID to be used. That patch still needs
a Signed-off-by from its original author, who once claimed not to
remember writing it at all. But now we've fixed it, hopefully he'll
admit it now :)
David Woodhouse (8):
x86/apic/x2apic: Fix parallel handling of cluster_mask
cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU
x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
x86/smpboot: Split up native_cpu_up into separate phases and document them
x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup
x86/smpboot: Serialize topology updates for secondary bringup
Thomas Gleixner (1):
x86/smpboot: Support parallel startup of secondary CPUs
[dwoodhou@i7 linux-2.6]$ git diff --stat v5.17-rc2..share/parallel-5.17-part1
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/smp.h | 13 +-
arch/x86/include/asm/topology.h | 2 -
arch/x86/kernel/acpi/sleep.c | 1 +
arch/x86/kernel/apic/apic.c | 2 +-
arch/x86/kernel/apic/x2apic_cluster.c | 108 ++++++-----
arch/x86/kernel/cpu/common.c | 6 +-
arch/x86/kernel/cpu/mtrr/mtrr.c | 9 +
arch/x86/kernel/head_64.S | 73 ++++++++
arch/x86/kernel/smpboot.c | 325 ++++++++++++++++++++++++----------
arch/x86/realmode/init.c | 3 +
arch/x86/realmode/rm/trampoline_64.S | 14 ++
arch/x86/xen/smp_pv.c | 4 +-
include/linux/cpuhotplug.h | 2 +
include/linux/smpboot.h | 7 +
kernel/cpu.c | 27 ++-
kernel/smpboot.c | 2 +-
kernel/smpboot.h | 2 -
18 files changed, 442 insertions(+), 161 deletions(-)
