On Thu, Feb 27, 2025 at 11:32:26AM +0900, Honggyu Kim wrote:
> Actually, we're aware of this issue and currently trying to fix this.
> In our system, we've attached 4ch of CXL memory for each socket as
> follows.
>
> node0 node1
> +-------+ UPI +-------+
> | CPU 0 |-+-----+-| CPU 1 |
> +-------+ +-------+
> | DRAM0 | | DRAM1 |
> +---+---+ +---+---+
> | |
> +---+---+ +---+---+
> | CXL 0 | | CXL 4 |
> +---+---+ +---+---+
> | CXL 1 | | CXL 5 |
> +---+---+ +---+---+
> | CXL 2 | | CXL 6 |
> +---+---+ +---+---+
> | CXL 3 | | CXL 7 |
> +---+---+ +---+---+
> node2 node3
>
> The 4ch of CXL memory are detected as a single NUMA node in each socket,
> but it shows as follows with the current N_POSSIBLE loop.
>
> $ ls /sys/kernel/mm/mempolicy/weighted_interleave/
> node0 node1 node2 node3 node4 node5
> node6 node7 node8 node9 node10 node11
This is insufficient information for me to assess the correctness of the
configuration. Can you please show the contents of your CEDT/CFMWS and
SRAT/Memory Affinity structures?
mkdir acpi_data && cd acpi_data
acpidump -b
iasl -d *
cat cedt.dsl <- find all CFMWS entries
cat srat.dsl <- find all Memory Affinity entries
Basically I need to know:
1) Is each CXL device on a dedicated Host Bridge?
2) Is inter-host-bridge interleaving configured?
3) Is intra-host-bridge interleaving configured?
4) Do SRAT entries exist for all nodes?
5) Why are there 12 nodes but only 10 sources? Are there additional
devices left out of your diagram? Are there 2 CFMWS but and 8 Memory
Affinity records - resulting in 10 nodes? This is strange.
By default, Linux creates a node for each proximity domain ("PXM")
detected in the SRAT Memory Affinity tables. If SRAT entries for a
memory region described in a CFMWS is absent, it will also create an
node for that CFMWS.
Your reported configuration and results lead me to believe you have
a combination of CFMWS/SRAT configurations that are unexpected.
~Gregory
