Wido, I am going to put my rather large foot in it here.
I am sure it is understood that the Turbo mode will not keep all cores at the maximum frequency at any given time.
There is a thermal envelope for the chip, and the chip works to keep the power dissipation within that envelope.
From what I gather there is a range of thermal limits even within a given processor SKU, so every chip will exhibit
different Turbo mode behaviour.
And I am sure we all know that when AVX comes into use the Turbo limit is lower.
I guess what I am saying that for to have reproducible behaviour, if you care about it for timings etc. Turbo
can be switched off.
Before you say it, in this case you want to achieve the minimum latency and reproducibility at the Mhz level is not important.
Also worth saying that cooling is important with Turboboost comes into play. I heard a paper at an HPC Advisory Council
where a Russian setup by Lenovo got significantly more performance at the HPC acceptance testing stage when cooling was turned up.
I guess my rambling has not added much to this debate, sorry.
cue a friendly Intel engineer to wander in and tell us exactly what is going on.
On 14 May 2018 at 15:13, Wido den Hollander <wido@xxxxxxxx> wrote:
On 05/01/2018 10:19 PM, Nick Fisk wrote:
> 4.16 required?
> https://www.phoronix.com/scan.php?page=news_item&px=Skylake- X-P-State-Linux-
> 4.16
>
I've been trying with the 4.16 kernel for the last few days, but still,
it's not working.
The CPU's keep clocking down to 800Mhz
I've set scaling_min_freq=scaling_max_freq in /sys, but that doesn't
change a thing. The CPUs keep scaling down.
Still not close to the 1ms latency with these CPUs :(
Wido
>
> -----Original Message-----
> From: ceph-users <ceph-users-bounces@lists.ceph.com > On Behalf Of Blair
> Bethwaite
> Sent: 01 May 2018 16:46
> To: Wido den Hollander <wido@xxxxxxxx>
> Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx>; Nick Fisk <nick@xxxxxxxxxx>
> Subject: Re: Intel Xeon Scalable and CPU frequency scaling on
> NVMe/SSD Ceph OSDs
>
> Also curious about this over here. We've got a rack's worth of R740XDs with
> Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active on them,
> though I don't believe they are any different at the OS level to our
> Broadwell nodes (where it is loaded).
>
> Have you tried poking the kernel's pmqos interface for your use-case?
>
> On 2 May 2018 at 01:07, Wido den Hollander <wido@xxxxxxxx> wrote:
>> Hi,
>>
>> I've been trying to get the lowest latency possible out of the new
>> Xeon Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
>>
>> However, I can't seem to pin the CPUs to always run at their maximum
>> frequency.
>>
>> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver
>> 4110), but that disables the boost.
>>
>> With the Power Saving enabled in the BIOS and when giving the OS all
>> control for some reason the CPUs keep scaling down.
>>
>> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>
>> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report
>> errors and bugs to cpufreq@xxxxxxxxxxxxxxx, please.
>> analyzing CPU 0:
>> driver: intel_pstate
>> CPUs which run at the same hardware frequency: 0
>> CPUs which need to have their frequency coordinated by software: 0
>> maximum transition latency: 0.97 ms.
>> hardware limits: 800 MHz - 3.00 GHz
>> available cpufreq governors: performance, powersave
>> current policy: frequency should be within 800 MHz and 3.00 GHz.
>> The governor "performance" may decide which speed to use
>> within this range.
>> current CPU frequency is 800 MHz.
>>
>> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down
>> again to 800Mhz and that hurts latency. (50% difference!)
>>
>> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to
>> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
>>
>> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
>> performance
>>
>> Everything seems to be OK and I would expect the CPUs to stay at
>> 2.10Ghz, but they aren't.
>>
>> C-States are also pinned to 0 as a boot parameter for the kernel:
>>
>> processor.max_cstate=1 intel_idle.max_cstate=0
>>
>> Running Ubuntu 16.04.4 with the 4.13 kernel from the HWE from Ubuntu.
>>
>> Has anybody tried this yet with the recent Intel Xeon Scalable CPUs?
>>
>> Thanks,
>>
>> Wido
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>
>
>
> --
> Cheers,
> ~Blairo
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
