On Fri, May 12, 2023 at 11:07:50PM +0200, Thomas Gleixner wrote: > From: Thomas Gleixner <tglx@xxxxxxxxxxxxx> > > There is often significant latency in the early stages of CPU bringup, and > time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and > then waiting for it to respond before moving on to the next. > > Allow a platform to enable parallel setup which brings all to be onlined > CPUs up to the CPUHP_BP_KICK_AP state. While this state advancement on the > control CPU (BP) is single-threaded the important part is the last state > CPUHP_BP_KICK_AP which wakes the to be onlined CPUs up. We're seeing a regression on ThunderX2 systems with 256 CPUs with an arm64 defconfig running -next which I've bisected to this patch. Before this commit we bring up 256 CPUs: [ 29.137225] GICv3: CPU254: found redistributor 11e03 region 1:0x0000000441f60000 [ 29.137238] GICv3: CPU254: using allocated LPI pending table @0x00000008818e0000 [ 29.137305] CPU254: Booted secondary processor 0x0000011e03 [0x431f0af1] [ 29.292421] Detected PIPT I-cache on CPU255 [ 29.292635] GICv3: CPU255: found redistributor 11f03 region 1:0x0000000441fe0000 [ 29.292648] GICv3: CPU255: using allocated LPI pending table @0x00000008818f0000 [ 29.292715] CPU255: Booted secondary processor 0x0000011f03 [0x431f0af1] [ 29.292859] smp: Brought up 2 nodes, 256 CPUs [ 29.292864] SMP: Total of 256 processors activated. but after we only bring up 255, missing the 256th: [ 29.165888] GICv3: CPU254: found redistributor 11e03 region 1:0x0000000441f60000 [ 29.165901] GICv3: CPU254: using allocated LPI pending table @0x00000008818e0000 [ 29.165968] CPU254: Booted secondary processor 0x0000011e03 [0x431f0af1] [ 29.166120] smp: Brought up 2 nodes, 255 CPUs [ 29.166125] SMP: Total of 255 processors activated. I can't immediately see an issue with the patch itself, for systems without CONFIG_HOTPLUG_PARALLEL=y it should replace the loop over cpu_present_mask done by for_each_present_cpu() with an open coded one. I didn't check the rest of the series yet. The KernelCI bisection bot also isolated an issue on Odroid XU3 (a 32 bit arm system) with the final CPU of the 8 on the system not coming up to the same patch: https://groups.io/g/kernelci-results/message/42480?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2Call-cpus%2C20%2C2%2C0%2C99054444 Other boards I've checked (including some with multiple CPU clusters) seem to be bringing up all their CPUs so it doesn't seem to just be general breakage. Log from my bisect: git bisect start # bad: [9f258af06b6268be8e960f63c3f66e88bdbbbdb0] Add linux-next specific files for 20230522 git bisect bad 9f258af06b6268be8e960f63c3f66e88bdbbbdb0 # good: [44c026a73be8038f03dbdeef028b642880cf1511] Linux 6.4-rc3 git bisect good 44c026a73be8038f03dbdeef028b642880cf1511 # good: [914db90ee0172753ab5298a48c63ac4f1fe089cf] Merge branch 'for-linux-next' of git://anongit.freedesktop.org/drm/drm-misc git bisect good 914db90ee0172753ab5298a48c63ac4f1fe089cf # good: [4624865b65777295cbe97cf1b98e6e49d81119d3] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input.git git bisect good 4624865b65777295cbe97cf1b98e6e49d81119d3 # bad: [be7220c44fbc06825f7f122d06051630e1bf51e4] Merge branch 'for-next' of git://github.com/cminyard/linux-ipmi.git git bisect bad be7220c44fbc06825f7f122d06051630e1bf51e4 # good: [cc677f7bec0da862a93d176524cdad5f416d58ef] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git git bisect good cc677f7bec0da862a93d176524cdad5f416d58ef # bad: [cdcc744aee1b886cbe4737798c0b8178b9ba5ae5] next-20230518/rcu git bisect bad cdcc744aee1b886cbe4737798c0b8178b9ba5ae5 # bad: [8397dce1586a35af63fe9ea3e8fb3344758e55b5] Merge branch into tip/master: 'x86/mm' git bisect bad 8397dce1586a35af63fe9ea3e8fb3344758e55b5 # bad: [0c7ffa32dbd6b09a87fea4ad1de8b27145dfd9a6] x86/smpboot/64: Implement arch_cpuhp_init_parallel_bringup() and enable it git bisect bad 0c7ffa32dbd6b09a87fea4ad1de8b27145dfd9a6 # good: [ab24eb9abb9c60c45119370731735b79ed79f36c] x86/xen/hvm: Get rid of DEAD_FROZEN handling git bisect good ab24eb9abb9c60c45119370731735b79ed79f36c # good: [72b11aa7f8f93449141544cecb21b2963416902d] riscv: Switch to hotplug core state synchronization git bisect good 72b11aa7f8f93449141544cecb21b2963416902d # good: [f54d4434c281f38b975d58de47adeca671beff4f] x86/apic: Provide cpu_primary_thread mask git bisect good f54d4434c281f38b975d58de47adeca671beff4f # bad: [bea629d57d006733d155bdb65ba4867788da69b6] x86/apic: Save the APIC virtual base address git bisect bad bea629d57d006733d155bdb65ba4867788da69b6 # bad: [18415f33e2ac4ab382cbca8b5ff82a9036b5bd49] cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE git bisect bad 18415f33e2ac4ab382cbca8b5ff82a9036b5bd49 # first bad commit: [18415f33e2ac4ab382cbca8b5ff82a9036b5bd49] cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE
Attachment:
signature.asc
Description: PGP signature
