On Mon, 28 Oct 2019 at 16:25, Rob Herring <robh@xxxxxxxxxx> wrote: > > On Mon, Oct 28, 2019 at 6:00 AM Francois Ozog <francois.ozog@xxxxxxxxxx> wrote: > > > > (reposting because of HTML mail format... sorry) > > > > > > On Sat, 26 Oct 2019 at 22:32, Olof Johansson <olof@xxxxxxxxx> wrote: > > > > > > On Sat, Oct 26, 2019 at 5:12 AM Francois Ozog <francois.ozog@xxxxxxxxxx> wrote: > > > > > > > > Hi, > > > > > > > > I'd like to share some past experience that may be relevant to the SDT > > > > discussion. > > > > > > > > In the context of 10Gbps networking I started to work on memory > > > > affinity back in 2005. At some point I observed a processor with 16 > > > > cores and 4 memory channels, organized internally on two > > > > interconnected dual rings (8 cores + 2 memory channels on a single > > > > dual ring). > > > > If you assign memory on the wrong dual ring, you have a 30% or more > > > > performance penalty. Interleaving at various stages (socket, channel, > > > > rank...) is not helping because we try to keep the hot data set as > > > > small as possible (granules for interleaving were 64MB or 128 bytes > > > > depending on the level and selected decoder policies that could not be > > > > changed despite programmable). > > > > > > This is literally what the DT numa spec already describes, isn't it? > > > > > > https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/numa.txt > > > > > On a Xeon, even a 5 years old one, there can be 2 proximity domains On > > a single socket. > > So if the numa node can represent that then the text shall be enhanced > > to actually capture that a single socket can have more than one numa > > node depending on the architecture . > > The example says 2 sockets, but that is completely outside the scope > of the binding. IOW, a 16 core SoC with 2 domains would have exactly > the same binding. > > > > Interleaving indeed counteracts any efforts on describing topology if > > > you interleave between different entities. > > > > > > > Some "good" ACPI systems where properly reporting the distances > > > > between the cores and the memory channels, with visible increased > > > > cost if you use wrong proximity domain. So advanced programmers were > > > > able to leverage the topology at its best ;-) > > > > > > > > Some technology appear to protect L3 cache for certain VMs and with > > > > more sensitivity on latency and jitter I would guess that capturing > > > > the right topology shall become (is becoming?) a priority. > > > > > > > > Too bad, Linux NUMA policy completely masks the intra-socket > > > > asymmetry. Taking into HPC, CCIX and CXL, the different memory > > > > hierarchies may need a way richer information set than just the NUMA > > > > socket. > > > > > > There's no restriction on NUMA policy being bound only at the unit of > > > a socket, you can choose to define domains as you see fit. The same > > > challenges apply to some of the modern x86 platforms such as AMD's > > > multi-die chips where some CPU chiplets have memory close to them and > > > others don't. > > > > > The Documentation text loosely describes two cases and each case is > > bound to socket limits. Too bad then. > > Sorry, I don't follow. Do you have an example you don't think is > covered by the binding? > Looks like I based my understanding on the fact that some older Xeon processors should have exposed two numa nodes per socket and thy did not for some obscure reason. I think I now properly understand the model and am happy with it. I guess it may be nice to have some explicit statement about numa node vs processor socket in different parts of the documentation: firmware (ACPI/DT) and Linux. > Rob -- François-Frédéric Ozog | Director Linaro Edge & Fog Computing Group T: +33.67221.6485 francois.ozog@xxxxxxxxxx | Skype: ffozog
