> > Intel server systems starting with Skylake support a mode that logically > > partitions each socket. E.g. when partitioned two ways, half the cores, > > L3 cache, and memory controllers are allocated to each of the partitions. > > This may reduce average latency to access L3 cache and memory, with the > > tradeoff that only half the L3 cache is available for subnode-local memory > > access. > > I couldn't find a description of what happens to the CAT bitmaps or counters. No changes to CAT. The cache is partitioned between sub-numa nodes based on the index, not by dividing the ways. E.g. an 8-way associative 32MB cache is still 8-way associative in each sub-node, but with 16MB available to each node. This means users who want a specific amount of cache will need to allocate more bits in the cache way mask (because each way is half as big). > Presumably the CAT bitmaps are duplicated, so each cluster has its own set, and > the counters aren't - so software has to co-ordinate the use of RMID across the CPUs? Nope. Still one set of CAT bit maps per socket. With "N" RMIDs available on a system with SNC disabled, there will be N/2 available when there are 2 SNC nodes per socket. Processes use values [0 .. N/2). > How come cacheinfo isn't modified to report the L3 partitions as separate caches? > Otherwise user-space would assume the full size of the cache is available on any of those > CPUs. The size of the cache is perhaps poorly defined in the SNC enabled case. A well behaved NUMA application that is only accessing memory from its local node will see an effective cache half the size. But if a process accesses memory from the other SNC node on the same socket, then it will get allocations in that SNC nodes half share of the cache. Accessing memory across inter-socket links will end up allocating across the whole cache. Moral: SNC mode is intended for applications that have very well-behaved NUMA characteristics. -Tony
