On Sun, Mar 5, 2023 at 10:17 AM Linus Torvalds
<torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:
>
> There are a few main issues with MAXSMP:
It's probably worth noting that most architectures don't even support
MAXSMP at all.
Only x86-64 does.
For example, ia64 and sparc64, which both did techncially support a
lot of cores, just made "cpumask_t" huge, and had no support for the
whole "use a pointer to an indirect allocation".
That ends up meaning that you allocate those huge structures on the
stack or just make other structures enormous when they contain a CPU
mask, but it mostly works. It's a horrid, horrid model, though. But at
least ia64 had 64kB stacks anyway, and in the book of "bad engineering
decisions of Itanium", this is all just a footnote.
arm64 also has that "range 2 4096" for number of CPUs but defaults to
a much saner 256 cpus.
I suspect (and sincerely hope) that nobody actually tries to use an
arm64 build with that 4k cpu build. If/when arm64 actually does get up
to that 'thousands of cores" situation, they'll hopefully enable the
MAXSMP kind of indirection and off-stack cpu mask arrays.
So MAXSMP and the whole CPUMASK_OFFSTACK option is an architecture
choice, and you don't have to do it the way x86-64 does it. But the
x86 choice is likely the best tested and thought out by far.
For example, POWERPC technically supports CPUMASK_OFFSTACK too, but
really only in theory. On powerpc, you have
config NR_CPUS
range 2 8192 if SMP
default "32" if PPC64
so while configuration the range is technically up to 8k CPUs, I doubt
people use that value very much. And we have
select CPUMASK_OFFSTACK if NR_CPUS >= 8192
so it only uses that OFFSTACK one if you pick exactly 8192 CPUs (which
presumably nobody does in real life outside of build testing - it's
not the default, and I think most of the POWER range tops up in the
192 core range, eg E980 with 16 sockets of 12 cores each).
So I suspect that x86-64 is the *only* one to actually use this
widely, and I think distros have been *much* too eager to do so.
The fact that most distros default to
CONFIG_MAXSMP=y
CONFIG_NR_CPUS=8192
seems pretty crazy, when I have a hard time finding anything with more
than 192 cores. I'm sure they exist. But do they _really_ run
unmodified vendor kernels?
Linus
