Hello Peter, On Fri, Jun 02, 2023 at 10:47:07AM +0530, K Prateek Nayak wrote: > Hello Peter, > > On 6/1/2023 8:21 PM, Peter Zijlstra wrote: > > On Thu, Jun 01, 2023 at 02:00:01PM +0200, Peter Zijlstra wrote: > >> On Thu, Jun 01, 2023 at 01:56:43PM +0200, Peter Zijlstra wrote: > >>> On Thu, Jun 01, 2023 at 01:13:26PM +0200, Peter Zijlstra wrote: > >>>> > >>>> This DeathStarBench thing seems to suggest that scanning up to 4 CCDs > >>>> isn't too much of a bother; so perhaps something like so? > >>>> > >>>> (on top of tip/sched/core from just a few hours ago, as I had to 'fix' > >>>> this patch and force pushed the thing) > >>>> > >>>> And yeah, random hacks and heuristics here :/ Does there happen to be > >>>> additional topology that could aid us here? Does the CCD fabric itself > >>>> have a distance metric we can use? > >>> > >>> https://www.anandtech.com/show/16529/amd-epyc-milan-review/4 > >>> > >>> Specifically: > >>> > >>> https://images.anandtech.com/doci/16529/Bounce-7763.png > >>> > >>> That seems to suggest there are some very minor distance effects in the > >>> CCD fabric. I didn't read the article too closely, but you'll note that > >>> the first 4 CCDs have inter-CCD latency < 100 while the rest has > 100. > >>> > >>> Could you also test on a Zen2 Epyc, does that require nr=8 instead of 4? > >>> Should we perhaps write it like: 32 / llc_size ? > >>> > >>> The Zen2 picture: > >>> > >>> https://images.anandtech.com/doci/16315/Bounce-7742.png > >>> > >>> Shows a more pronounced CCD fabric topology, you can really see the 2 > >>> CCX inside the CCD but also there's two ligher green squares around the > >>> CCDs themselves. > >> > >> I can't seem to find pretty pictures for Zen4 Epyc; what does that want? > >> That's even bigger at 96/8=12 LLCs afaict. > > > > Going by random pictures on the interweb again, it looks like this Zen4 > > thing wants either 2 groups of 6 each, or 4 groups of 3. > Yes, this is what the topology looks like |---------------------------------------------------------------------------------| | | | ----------- ----------- ----------- ----------- ----------- ----------- | | |(0-7) | |(8-15) | |(16-23) | |(48-55) | |(56-63) | |(64-71) | | | | LLC0 | | LLC1 | | LLC2 | | LLC6 | | LLC7 | | LLC8 | | | |(96-103) | |(104-111)| |(112-119)| |(144-151)| |(152-159)| |(160-167)| | | ----------- ----------- ----------- ----------- ----------- ----------- | | | | | | ----------- ----------- ----------- ----------- ----------- ----------- | | |(24-31) | |(32-39) | |(40-47) | |(72-79) | |(80-87) | |(88-95) | | | | LLC3 | | LLC4 | | LLC5 | | LLC9 | | LLC10 | | LLC11 | | | |(120-127)| |(128-135)| |(136-143)| |(168-175)| |(176-183)| |(184-191)| | | ----------- ----------- ----------- ----------- ----------- ----------- | | | |---------------------------------------------------------------------------------| > I would think it is the latter since NPS4 does that but let me go verify. 2 groups of 6 each is the vertical split which is NPS2. 4 groups of 3 each is the vertical and horizontal split, which is NPS4. In both these cases, currently the domain hierarchy SMT --> MC --> NODE --> NUMA where the NODE will be the parent of MC and be the 2nd level wakeup domain. If we define CLS to be the group with 3 LLCs, which becomes the parent of the MC domain, then, the hierarchy would be NPS1 : SMT --> MC --> CLS --> DIE NPS2 : SMT --> MC --> CLS --> NODE --> NUMA NPS4 : SMT --> MC --> CLS --> NUMA NPS2 will have 5 domains within a single socket. Oh well! -- Thanks and Regards gautham.
|