Dear All,
On 07.03.2024 02:45, Christoph Lameter (Ampere) wrote:
> Currently defconfig selects NR_CPUS=256, but some vendors (e.g. Ampere
> Computing) are planning to ship systems with 512 CPUs. So that all
> CPUs on
> these systems can be used with defconfig, we'd like to bump NR_CPUS to
> 512.
> Therefore this patch increases the default NR_CPUS from 256 to 512.
>
> As increasing NR_CPUS will increase the size of cpumasks, there's a
> fear that
> this might have a significant impact on stack usage due to code which
> places
> cpumasks on the stack. To mitigate that concern, we can select
> CPUMASK_OFFSTACK. As that doesn't seem to be a problem today with
> NR_CPUS=256, we only select this when NR_CPUS > 256.
>
> CPUMASK_OFFSTACK configures the cpumasks in the kernel to be
> dynamically allocated. This was used in the X86 architecture in the
> past to enable support for larger CPU configurations up to 8k cpus.
>
> With that is becomes possible to dynamically size the allocation of
> the cpu bitmaps depending on the quantity of processors detected on
> bootup. Memory used for cpumasks will increase if the kernel is
> run on a machine with more cores.
>
> Further increases may be needed if ARM processor vendors start
> supporting more processors. Given the current inflationary trends
> in core counts from multiple processor manufacturers this may occur.
>
> There are minor regressions for hackbench. The kernel data size
> for 512 cpus is smaller with offstack than with onstack.
>
> Benchmark results using hackbench average over 10 runs of
>
> hackbench -s 512 -l 2000 -g 15 -f 25 -P
>
> on Altra 80 Core
>
> Support for 256 CPUs on stack. Baseline
>
> 7.8564 sec
>
> Support for 512 CUs on stack.
>
> 7.8713 sec + 0.18%
>
> 512 CPUS offstack
>
> 7.8916 sec + 0.44%
>
> Kernel size comparison:
>
> text data filename Difference to
> onstack256 baseline
> 25755648 9589248 vmlinuz-6.8.0-rc4-onstack256
> 25755648 9607680 vmlinuz-6.8.0-rc4-onstack512 +0.19%
> 25755648 9603584 vmlinuz-6.8.0-rc4-offstack512 +0.14%
>
> Tested-by: Eric Mackay <eric.mackay@xxxxxxxxxx>
> Reviewed-by: Russell King (Oracle) <rmk+kernel@xxxxxxxxxxxxxxx>
> Signed-off-by: Christoph Lameter (Ampere) <cl@xxxxxxxxx>
This patch landed in today's linux-next as commit 0499a78369ad ("ARM64:
Dynamically allocate cpumasks and increase supported CPUs to 512").
Unfortunately it triggers the following warning during boot on most of
my ARM64-based test boards. Here is an example from Odroid-N2 board:
------------[ cut here ]------------
WARNING: CPU: 4 PID: 63 at drivers/opp/core.c:2554
dev_pm_opp_set_config+0x390/0x710
Modules linked in: dw_hdmi_i2s_audio meson_gxl smsc onboard_usb_hub(+)
rtc_pcf8563 panfrost snd_soc_meson_axg_sound_card drm_shmem_helper
crct10dif_ce dwmac_generic snd_soc_meson_card_utils gpu_sched
snd_soc_meson_g12a_toacodec snd_soc_meson_g12a_tohdmitx rc_odroid
snd_soc_meson_codec_glue pwm_meson ao_cec_g12a meson_gxbb_wdt meson_ir
snd_soc_meson_axg_frddr snd_soc_meson_axg_toddr snd_soc_meson_axg_tdmin
meson_vdec(C) v4l2_mem2mem videobuf2_dma_contig snd_soc_meson_axg_tdmout
videobuf2_memops axg_audio videobuf2_v4l2 sclk_div videodev
reset_meson_audio_arb snd_soc_meson_axg_fifo clk_phase dwmac_meson8b
stmmac_platform videobuf2_common mdio_mux_meson_g12a meson_drm
meson_dw_hdmi rtc_meson_vrtc stmmac meson_ddr_pmu_g12 mc dw_hdmi
drm_display_helper pcs_xpcs snd_soc_meson_t9015 meson_canvas gpio_fan
display_connector snd_soc_meson_axg_tdm_interface
snd_soc_simple_amplifier snd_soc_meson_axg_tdm_formatter nvmem_meson_efuse
hub 1-1:1.0: USB hub found
CPU: 4 PID: 63 Comm: kworker/u12:5 Tainted: G C 6.8.0-rc3+ #14677
Hardware name: Hardkernel ODROID-N2 (DT)
Workqueue: events_unbound deferred_probe_work_func
pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : dev_pm_opp_set_config+0x390/0x710
lr : dev_pm_opp_set_config+0x5c/0x710
...
Call trace:
dev_pm_opp_set_config+0x390/0x710
dt_cpufreq_probe+0x268/0x480
platform_probe+0x68/0xd8
really_probe+0x148/0x2b4
__driver_probe_device+0x78/0x12c
driver_probe_device+0xdc/0x164
__device_attach_driver+0xb8/0x138
bus_for_each_drv+0x84/0xe0
__device_attach+0xa8/0x1b0
device_initial_probe+0x14/0x20
bus_probe_device+0xb0/0xb4
deferred_probe_work_func+0x8c/0xc8
process_one_work+0x1ec/0x53c
worker_thread+0x298/0x408
kthread+0x124/0x128
ret_from_fork+0x10/0x20
irq event stamp: 317178
hardirqs last enabled at (317177): [<ffff8000801788d4>]
ktime_get_coarse_real_ts64+0x10c/0x110
hardirqs last disabled at (317178): [<ffff800081222030>] el1_dbg+0x24/0x8c
softirqs last enabled at (315802): [<ffff800080010a60>]
__do_softirq+0x4a0/0x4e8
softirqs last disabled at (315793): [<ffff8000800169b0>]
____do_softirq+0x10/0x1c
---[ end trace 0000000000000000 ]---
cpu cpu2: error -EBUSY: failed to set regulators
cpufreq-dt: probe of cpufreq-dt failed with error -16
It looks that cpufreq-dt and/or opp drivers needs some adjustments
related with this change.
> ---
>
>
> Original post: https://www.spinics.net/lists/linux-mm/msg369701.html
> V2: https://lkml.org/lkml/2024/2/7/505
>
>
> V1->V2
>
> - Keep quotation marks
> - Remove whiltespace damage
> - Add tested by
>
> V2->V3:
> - Add test results
> - Rework descriptions
>
>
> arch/arm64/Kconfig | 16 +++++++++++++++-
> 1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index aa7c1d435139..4e767dede47d 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1427,7 +1427,21 @@ config SCHED_SMT
> config NR_CPUS
> int "Maximum number of CPUs (2-4096)"
> range 2 4096
> - default "256"
> + default "512"
> +
> +#
> +# Determines the placement of cpumasks.
> +#
> +# With CPUMASK_OFFSTACK the cpumasks are dynamically allocated.
> +# Useful for machines with lots of core because it avoids increasing
> +# the size of many of the data structures in the kernel.
> +#
> +# If this is off then the cpumasks have a static sizes and are
> +# embedded within data structures.
> +#
> + config CPUMASK_OFFSTACK
> + def_bool y
> + depends on NR_CPUS > 256
>
> config HOTPLUG_CPU
> bool "Support for hot-pluggable CPUs"
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
