Sorry, bit late to get back to this...
On Wed., 2 May 2018, 06:19 Nick Fisk, <nick@xxxxxxxxxx> wrote:
4.16 required?
Looks like it - thanks for pointing that out.
Wido, I don't think you are doing anything wrong here, maybe this is a bug...
I've got RHEL7 + Broadwell based Ceph nodes here for which the same tuning appears to be working fine:
-bash-4.2$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release: 7.3
Codename: Maipo
-bash-4.2$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 20
On-line CPU(s) list: 0-19
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Stepping: 1
CPU MHz: 2745.960
BogoMIPS: 4399.83
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-19
-bash-4.2$ cat /proc/cmdline
-bash-4.2$ sudo cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 1.20 GHz - 3.10 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 1.20 GHz and 3.10 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: 2.40 GHz (asserted by call to hardware)
boost state support:
Supported: yes
Active: yes
-bash-4.2$ sudo cpupower -c 0-19 monitor
|Nehalem || Mperf || Idle_Stats
CPU | C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1-B | C1E- | C3-B | C6-B
0| 0.00| 0.00| 0.00| 0.00|| 20.93| 79.07| 2398|| 1.00| 79.08| 0.00| 0.00| 0.00
10| 0.00| 0.00| 0.00| 0.00|| 1.81| 98.19| 2398|| 0.00| 98.23| 0.00| 0.00| 0.00
1| 0.00| 0.00| 0.00| 0.00|| 3.80| 96.20| 2398|| 2.10| 96.21| 0.00| 0.00| 0.00
11| 0.00| 0.00| 0.00| 0.00|| 7.95| 92.05| 2398|| 7.59| 92.06| 0.00| 0.00| 0.00
2| 0.00| 0.00| 0.00| 0.00|| 1.99| 98.01| 2398|| 0.00| 98.04| 0.00| 0.00| 0.00
12| 0.00| 0.00| 0.00| 0.00|| 1.59| 98.41| 2398|| 0.64| 98.42| 0.00| 0.00| 0.00
3| 0.00| 0.00| 0.00| 0.00|| 24.58| 75.42| 2398|| 0.00| 75.43| 0.00| 0.00| 0.00
13| 0.00| 0.00| 0.00| 0.00|| 1.66| 98.34| 2399|| 0.24| 98.35| 0.00| 0.00| 0.00
4| 0.00| 0.00| 0.00| 0.00|| 1.36| 98.64| 2398|| 0.00| 98.65| 0.00| 0.00| 0.00
14| 0.00| 0.00| 0.00| 0.00|| 1.95| 98.05| 2398|| 0.77| 98.06| 0.00| 0.00| 0.00
5| 0.00| 0.00| 0.00| 0.00|| 1.39| 98.61| 2398|| 0.00| 98.64| 0.00| 0.00| 0.00
15| 0.00| 0.00| 0.00| 0.00|| 8.33| 91.67| 2398|| 7.80| 91.68| 0.00| 0.00| 0.00
6| 0.00| 0.00| 0.00| 0.00|| 1.48| 98.52| 2398|| 0.00| 98.54| 0.00| 0.00| 0.00
16| 0.00| 0.00| 0.00| 0.00|| 2.44| 97.56| 2398|| 1.73| 97.57| 0.00| 0.00| 0.00
7| 0.00| 0.00| 0.00| 0.00|| 2.13| 97.87| 2398|| 0.64| 97.88| 0.00| 0.00| 0.00
17| 0.00| 0.00| 0.00| 0.00|| 1.03| 98.97| 2398|| 0.24| 98.93| 0.00| 0.00| 0.00
8| 0.00| 0.00| 0.00| 0.00|| 1.43| 98.57| 2398|| 0.00| 98.61| 0.00| 0.00| 0.00
18| 0.00| 0.00| 0.00| 0.00|| 1.58| 98.42| 2398|| 0.00| 98.45| 0.00| 0.00| 0.00
9| 0.00| 0.00| 0.00| 0.00|| 1.66| 98.34| 2398|| 0.00| 98.35| 0.00| 0.00| 0.00
19| 0.00| 0.00| 0.00| 0.00|| 1.04| 98.96| 2398|| 0.00| 98.93| 0.00| 0.00| 0.00
-bash-4.2$ sudo /opt/dell/srvadmin/bin/omreport chassis biossetup | egrep -i "c state|turbo"
Dell Controlled Turbo : Disabled
Turbo Boost : Enabled
Energy Efficient Turbo : Disabled
C States : Disabled
Number of Turbo Boost Enabled Cores for Processor 1 : All
-bash-4.2$ sudo tail /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq | grep ^2
2399976
2399890
2399976
2399976
2399976
2399804
2399976
2399976
2400062
2399976
2399976
2399890
2399976
2400062
2399976
2399976
2399804
2399890
2399976
2399890
We didn't manage to get this level of consistency until we used /dev/cpu_dma_latency (see https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt) via tuned:
-bash-4.2$ sudo tuned-adm active
Current active profile: latency-performance
We've got some Skylake Ubuntu based hypervisors that we can look at to compare tomorrow...
Cheers,
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
