The Sub-NUMA cluster feature on some Intel processors partitions the CPUs that share an L3 cache into two or more sets. This plays havoc with the Resource Director Technology (RDT) monitoring features. Prior to this patch Intel has advised that SNC and RDT are incompatible. Some of these CPU support an MSR that can partition the RMID counters in the same way. This allows monitoring features to be used. With the caveat that users must be aware that Linux may migrate tasks more frequently between SNC nodes than between "regular" NUMA nodes, so reading counters from all SNC nodes may be needed to get a complete picture of activity for tasks. Cache and memory bandwidth allocation features continue to operate at the scope of the L3 cache. Signed-off-by: Tony Luck <tony.luck@xxxxxxxxx> --- Changes since v14: https://lore.kernel.org/all/20240126223837.21835-1-tony.luck@xxxxxxxxx/ 1) Rebase to TIP x86/cache + my 2-patch cleanup 2) Dropped all Reviewed/Tested tags (enough changed in TIP that this needs looking at, and testing, again). 3) Added ATOM_CRESTMONT_X (Sierra Forest) to list of SNC supporting CPUs. 4) Added a console "INFO" message when SNC detected: pr_info("Sub-NUMA cluster detected with %d nodes per L3 cache\n", ret); Note that my alternate approach posted as v15-RFC https://lore.kernel.org/all/20240130222034.37181-1-tony.luck@xxxxxxxxx/ has been abandoned. I'd still like to explore splitting the L3 rdt_resource into separate control/monitor elements, but that can wait for another series of patches. Tony Luck (8): x86/resctrl: Prepare for new domain scope x86/resctrl: Prepare to split rdt_domain structure x86/resctrl: Prepare for different scope for control/monitor operations x86/resctrl: Split the rdt_domain and rdt_hw_domain structures x86/resctrl: Add node-scope to the options for feature scope x86/resctrl: Introduce snc_nodes_per_l3_cache x86/resctrl: Sub NUMA Cluster detection and enable x86/resctrl: Update documentation with Sub-NUMA cluster changes Documentation/arch/x86/resctrl.rst | 25 +- include/linux/resctrl.h | 85 +++-- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 67 ++-- arch/x86/kernel/cpu/resctrl/core.c | 428 ++++++++++++++++++---- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 56 +-- arch/x86/kernel/cpu/resctrl/monitor.c | 70 ++-- arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 26 +- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 156 ++++---- 9 files changed, 642 insertions(+), 272 deletions(-) -- 2.43.0
