Hi Gregory,
On Mon, 10 Mar 2025 22:42:52 -0400 Gregory Price <gourry@xxxxxxxxxx> wrote:
> On Tue, Mar 11, 2025 at 11:07:27AM +0900, Yunjeong Mun wrote:
> > Hi Gregory,
> >
> > In my understanding, both SRAT and CFMWS have the above device and interleave setup.
> >
> > and below are the CFMWS configurations (with some unnecessary lines removed):
> >
> > [0A4h 0164 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
> > [0ACh 0172 008h] Window base address : 0000002080000000 <- Memory region
> > [0B4h 0180 008h] Window size : 0000032780000000
> > [0BCh 0188 001h] Interleave Members : 01 <-- 2-way interleave
> > [0BDh 0189 001h] Interleave Arithmetic : 01
> > [0C8h 0200 004h] First Target : 00000043 <-- host bridge id
> > [0CCh 0204 004h] Next Target : 00000053 <-- host bridge id
> >
> > ...
> >
> > [170h 0368 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
> > [178h 0376 008h] Window base address : 0000064F80000000
> > [180h 0384 008h] Window size : 0000033000000000
> > [188h 0392 001h] Interleave Members : 01 <-- 2-way interleave
> > [189h 0393 001h] Interleave Arithmetic : 01
> > [194h 0404 004h] First Target : 00000143 <-- host bridge id
> > [198h 0408 004h] Next Target : 00000153 <-- host bridge id
> >
>
> Are you able to share all CXL Fixed Memory Window Structures in the
> CEDT? Just want to confirm some suspicions here about why we're seeing
> 12 NUMA nodes. This explains 2 and suggests there's at least 4 host
> bridges - but not the other 8.
>
Ok, there are 10 CFMWS entries and 4 host bridges in cedt.dsl.
Below is the information for all CFMWS entries and host bridge in the CEDT:
[024h 0036 001h] Subtable Type : 00 [CXL Host Bridge Structure]
[028h 0040 004h] Associated host bridge : 00000043
[044h 0068 001h] Subtable Type : 00 [CXL Host Bridge Structure]
[048h 0072 004h] Associated host bridge : 00000053
[064h 0100 001h] Subtable Type : 00 [CXL Host Bridge Structure]
[068h 0104 004h] Associated host bridge : 00000143
[084h 0132 001h] Subtable Type : 00 [CXL Host Bridge Structure]
[088h 0136 004h] Associated host bridge : 00000153
[0A4h 0164 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[0ACh 0172 008h] Window base address : 0000002080000000
[0B4h 0180 008h] Window size : 0000032780000000
[0BCh 0188 001h] Interleave Members : 01
[0BDh 0189 001h] Interleave Arithmetic : 01
[0C8h 0200 004h] First Target : 00000043
[0CCh 0204 004h] Next Target : 00000053
[0D0h 0208 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[0D8h 0216 008h] Window base address : 0000034800000000
[0E0h 0224 008h] Window size : 00000163C0000000
[0E8h 0232 001h] Interleave Members : 00
[0F4h 0244 004h] First Target : 00000043
[0F8h 0248 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[100h 0256 008h] Window base address : 000004ABC0000000
[108h 0264 008h] Window size : 00000163C0000000
[110h 0272 001h] Interleave Members : 00
[111h 0273 001h] Interleave Arithmetic : 00
[11Ch 0284 004h] First Target : 00000053
[120h 0288 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[128h 0296 008h] Window base address : 00000C2F80000000
[130h 0304 008h] Window size : 000002F840000000
[138h 0312 001h] Interleave Members : 00
[139h 0313 001h] Interleave Arithmetic : 00
[144h 0324 004h] First Target : 00000043
[148h 0328 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[150h 0336 008h] Window base address : 00000F27C0000000
[158h 0344 008h] Window size : 000002F840000000
[160h 0352 001h] Interleave Members : 00
[161h 0353 001h] Interleave Arithmetic : 00
[16Ch 0364 004h] First Target : 00000053
[170h 0368 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[178h 0376 008h] Window base address : 0000064F80000000
[180h 0384 008h] Window size : 0000033000000000
[188h 0392 001h] Interleave Members : 01
[189h 0393 001h] Interleave Arithmetic : 01
[194h 0404 004h] First Target : 00000143
[198h 0408 004h] Next Target : 00000153
[19Ch 0412 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[1A4h 0420 008h] Window base address : 0000097F80000000
[1ACh 0428 008h] Window size : 0000015800000000
[1B4h 0436 001h] Interleave Members : 00
[1B5h 0437 001h] Interleave Arithmetic : 00
[1C0h 0448 004h] First Target : 00000143
[1C4h 0452 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[1CCh 0460 008h] Window base address : 00000AD780000000
[1D4h 0468 008h] Window size : 0000015800000000
[1DCh 0476 001h] Interleave Members : 00
[1DDh 0477 001h] Interleave Arithmetic : 00
[1E8h 0488 004h] First Target : 00000153
[1ECh 0492 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[1F4h 0500 008h] Window base address : 0000122000000000
[1FCh 0508 008h] Window size : 000002F000000000
[204h 0516 001h] Interleave Members : 00
[205h 0517 001h] Interleave Arithmetic : 00
[210h 0528 004h] First Target : 00000143
[214h 0532 001h] Subtable Type : 01 [CXL Fixed Memory Window Structure]
[21Ch 0540 008h] Window base address : 0000151000000000
[224h 0548 008h] Window size : 000002F000000000
[22Ch 0556 001h] Interleave Members : 00
[22Dh 0557 001h] Interleave Arithmetic : 00
[238h 0568 004h] First Target : 00000153
In my understanding, the reason we are seeing 12 NUMA node is because
it loops through node_states[N_POSSIBLE] and its value is 4095 (twelves ones)
in the code [1] below:
for_each_node_state(nid, N_POSSIBLE) {
err = add_weight_node(nid, wi_kobj);
if (err) {
pr_err("failed to add sysfs [node%d]\n", nid);
break;
}
}
Since I'm not familiar with the CEDT, I'm wondering if you mean that N_POSSIBLE
is associated with the number of CFMWS.
[1] https://github.com/torvalds/linux/blob/v6.14-rc6/mm/mempolicy.c#L3508-L3514
> ~Gregory
>
Best regrads,
Yunjeong
