Re: Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 05/15/2018 02:51 PM, Blair Bethwaite wrote:
> Sorry, bit late to get back to this...
> 
> On Wed., 2 May 2018, 06:19 Nick Fisk, <nick@xxxxxxxxxx
> <mailto:nick@xxxxxxxxxx>> wrote:
> 
>     4.16 required?
> 
> 
> Looks like it - thanks for pointing that out.
> 
> Wido, I don't think you are doing anything wrong here, maybe this is a
> bug...
> 
> I've got RHEL7 + Broadwell based Ceph nodes here for which the same
> tuning appears to be working fine:
> 

Odd indeed. Keep in mind that I indeed have the newer Intel Scalable
CPUs with Ubuntu 16.04 and a 4.16 kernel.

My main goal is the lowest possible latency with NVMe and for that you
need higher clock speeds.

(more down)

> -bash-4.2$ lsb_release -a
> LSB Version:    :core-4.1-amd64:core-4.1-noarch
> Distributor ID: RedHatEnterpriseServer
> Description:    Red Hat Enterprise Linux Server release 7.3 (Maipo)
> Release:        7.3
> Codename:       Maipo
> 
> -bash-4.2$ lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                20
> On-line CPU(s) list:   0-19
> Thread(s) per core:    2
> Core(s) per socket:    10
> Socket(s):             1
> NUMA node(s):          1
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 79
> Model name:            Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:              1
> CPU MHz:               2745.960
> BogoMIPS:              4399.83
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              25600K
> NUMA node0 CPU(s):     0-19
> 
> -bash-4.2$ cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-3.10.0-514.2.2.el7.x86_64
> root=/dev/mapper/vg00-LogVol00 ro nofb splash=quiet crashkernel=auto
> rd.lvm.lv <http://rd.lvm.lv>=vg00/LogVol00 rd.lvm.lv
> <http://rd.lvm.lv>=vg00/LogVol01 rhgb quiet LANG=en_US.UTF-8
> 
> -bash-4.2$ sudo cpupower frequency-info
> analyzing CPU 0:
>   driver: intel_pstate
>   CPUs which run at the same hardware frequency: 0
>   CPUs which need to have their frequency coordinated by software: 0
>   maximum transition latency:  Cannot determine or is not supported.
>   hardware limits: 1.20 GHz - 3.10 GHz
>   available cpufreq governors: performance powersave
>   current policy: frequency should be within 1.20 GHz and 3.10 GHz.
>                   The governor "performance" may decide which speed to use
>                   within this range.
>   current CPU frequency: 2.40 GHz (asserted by call to hardware)
>   boost state support:
>     Supported: yes
>     Active: yes
> 
> -bash-4.2$ sudo cpupower -c 0-19 monitor
>     |Nehalem                    || Mperf              || Idle_Stats
> CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq || POLL | C1-B |
> C1E- | C3-B | C6-B
>    0|  0.00|  0.00|  0.00|  0.00|| 20.93| 79.07|  2398||  1.00| 79.08| 
> 0.00|  0.00|  0.00
>   10|  0.00|  0.00|  0.00|  0.00||  1.81| 98.19|  2398||  0.00| 98.23| 
> 0.00|  0.00|  0.00
>    1|  0.00|  0.00|  0.00|  0.00||  3.80| 96.20|  2398||  2.10| 96.21| 
> 0.00|  0.00|  0.00
>   11|  0.00|  0.00|  0.00|  0.00||  7.95| 92.05|  2398||  7.59| 92.06| 
> 0.00|  0.00|  0.00
>    2|  0.00|  0.00|  0.00|  0.00||  1.99| 98.01|  2398||  0.00| 98.04| 
> 0.00|  0.00|  0.00
>   12|  0.00|  0.00|  0.00|  0.00||  1.59| 98.41|  2398||  0.64| 98.42| 
> 0.00|  0.00|  0.00
>    3|  0.00|  0.00|  0.00|  0.00|| 24.58| 75.42|  2398||  0.00| 75.43| 
> 0.00|  0.00|  0.00
>   13|  0.00|  0.00|  0.00|  0.00||  1.66| 98.34|  2399||  0.24| 98.35| 
> 0.00|  0.00|  0.00
>    4|  0.00|  0.00|  0.00|  0.00||  1.36| 98.64|  2398||  0.00| 98.65| 
> 0.00|  0.00|  0.00
>   14|  0.00|  0.00|  0.00|  0.00||  1.95| 98.05|  2398||  0.77| 98.06| 
> 0.00|  0.00|  0.00
>    5|  0.00|  0.00|  0.00|  0.00||  1.39| 98.61|  2398||  0.00| 98.64| 
> 0.00|  0.00|  0.00
>   15|  0.00|  0.00|  0.00|  0.00||  8.33| 91.67|  2398||  7.80| 91.68| 
> 0.00|  0.00|  0.00
>    6|  0.00|  0.00|  0.00|  0.00||  1.48| 98.52|  2398||  0.00| 98.54| 
> 0.00|  0.00|  0.00
>   16|  0.00|  0.00|  0.00|  0.00||  2.44| 97.56|  2398||  1.73| 97.57| 
> 0.00|  0.00|  0.00
>    7|  0.00|  0.00|  0.00|  0.00||  2.13| 97.87|  2398||  0.64| 97.88| 
> 0.00|  0.00|  0.00
>   17|  0.00|  0.00|  0.00|  0.00||  1.03| 98.97|  2398||  0.24| 98.93| 
> 0.00|  0.00|  0.00
>    8|  0.00|  0.00|  0.00|  0.00||  1.43| 98.57|  2398||  0.00| 98.61| 
> 0.00|  0.00|  0.00
>   18|  0.00|  0.00|  0.00|  0.00||  1.58| 98.42|  2398||  0.00| 98.45| 
> 0.00|  0.00|  0.00
>    9|  0.00|  0.00|  0.00|  0.00||  1.66| 98.34|  2398||  0.00| 98.35| 
> 0.00|  0.00|  0.00
>   19|  0.00|  0.00|  0.00|  0.00||  1.04| 98.96|  2398||  0.00| 98.93| 
> 0.00|  0.00|  0.00
> 
> -bash-4.2$ sudo /opt/dell/srvadmin/bin/omreport chassis biossetup |
> egrep -i "c state|turbo"
> Dell Controlled Turbo                               : Disabled
> Turbo Boost                                         : Enabled
> Energy Efficient Turbo                              : Disabled
> C States                                            : Disabled
> Number of Turbo Boost Enabled Cores for Processor 1 : All
> 
> -bash-4.2$ sudo tail
> /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq | grep ^2
> 2399976
> 2399890
> 2399976
> 2399976
> 2399976
> 2399804
> 2399976
> 2399976
> 2400062
> 2399976
> 2399976
> 2399890
> 2399976
> 2400062
> 2399976
> 2399976
> 2399804
> 2399890
> 2399976
> 2399890
> 

This is what I want as well. My CPUs are 2.1Ghz and I want them to stay
there.

I've seen latencies of ~1ms when I manage to get the CPUs to stay high
clocked, but once they clock down my latency crawls back up to ~1.6ms.

> We didn't manage to get this level of consistency until we
> used /dev/cpu_dma_latency
> (see https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt) via
> tuned:
> 
> -bash-4.2$ sudo tuned-adm active
> Current active profile: latency-performance
> 
> We've got some Skylake Ubuntu based hypervisors that we can look at to
> compare tomorrow...
> 

Awesome!

> Cheers,
> 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux