I'd like to propose a discussion about the variety of CXL configuration
decisions made by platforms, BIOS, EFI, and linux that have created a
complex, and sometimes subtly confusing administration environment.
In particular, when and how memory configuration occurs have major
implications for major feature support (interleave, ras, hotplug, etc).
For example, treating CXL memory as "conventional" without marking is
"special purpose" may limit the applicability of certain RAS features -
but even marking it "special purpose" may be insufficient for
(coordinated) device hotplug compatibility.
Another example, RAS features like POISON have different end-state
implications (full system crash vs userland crash) that depend on
whether the memory was being used by kernel or userland (which is
actually somewhat controllable, and therefore useful for
administrators!)
Some of this complexity stems from interleave settings and how CXL
memory is distributed among NUMA nodes (1 per device, 1 per homogeneous
set, or a single heterogeneous numa node).
Specifically we'll talk about
- iomem resource allocation
- EFI_CONVENTIONAL_MEMORY, MEMORY_SP, and CONFIG_EFI_SOFT_RESERVE
- e820 & EFI mmemory map inclusion
- driver-time allocation
- hotplug implications
- Addressing
- SPA == HPA vs SPA != HPA
- Boot-time configuration vs Driver Configuration
- Interleave configuration
- Platform configuration vs Driver configuration
- PRMT-provided translation
- RAS feature implications
- Management implications (hotplug, teardown, etc)
- Linux Memory (Block) Hotplug
- auto-online vs user-policy
- systemd / typical user story
- Zone-assignment and Poison
I'd like to lay out (as best I can, with help!) the current environment in
linux kernel, the "maintenance implications" of certain configurations
decisions, and discuss where ambiguities are present / challenging.
I'll add some additional follow-on emails that break down some of these
scenarios more in-depth over the next few months for some background reading.
~Gregory
