Are these dual-socket machines? Perhaps NUMA is involved?
On Thu., Jun. 27, 2019, 4:56 a.m. Lars Täuber, <taeuber@xxxxxxx> wrote:
Hi!
In our cluster I ran some benchmarks.
The results are always similar but strange to me.
I don't know what the results mean.
The cluster consists of 7 (nearly) identical hosts for osds. Two of them have one an additional hdd.
The hdds are from identical type. The ssds for the journal and wal are of identical type. The configuration is identical (ssd-db-lv-size) for each osd.
The hosts are connected the same way to the same switches.
This nautilus cluster was set up with ceph-ansible 4.0 on debian buster.
This are the results of
# ceph --format plain tell osd.* bench
osd.0: bench: wrote 1 GiB in blocks of 4 MiB in 15.0133 sec at 68 MiB/sec 17 IOPS
osd.1: bench: wrote 1 GiB in blocks of 4 MiB in 6.98357 sec at 147 MiB/sec 36 IOPS
osd.2: bench: wrote 1 GiB in blocks of 4 MiB in 6.80336 sec at 151 MiB/sec 37 IOPS
osd.3: bench: wrote 1 GiB in blocks of 4 MiB in 12.0813 sec at 85 MiB/sec 21 IOPS
osd.4: bench: wrote 1 GiB in blocks of 4 MiB in 8.51311 sec at 120 MiB/sec 30 IOPS
osd.5: bench: wrote 1 GiB in blocks of 4 MiB in 6.61376 sec at 155 MiB/sec 38 IOPS
osd.6: bench: wrote 1 GiB in blocks of 4 MiB in 14.7478 sec at 69 MiB/sec 17 IOPS
osd.7: bench: wrote 1 GiB in blocks of 4 MiB in 12.9266 sec at 79 MiB/sec 19 IOPS
osd.8: bench: wrote 1 GiB in blocks of 4 MiB in 15.2513 sec at 67 MiB/sec 16 IOPS
osd.9: bench: wrote 1 GiB in blocks of 4 MiB in 9.26225 sec at 111 MiB/sec 27 IOPS
osd.10: bench: wrote 1 GiB in blocks of 4 MiB in 13.6641 sec at 75 MiB/sec 18 IOPS
osd.11: bench: wrote 1 GiB in blocks of 4 MiB in 13.8943 sec at 74 MiB/sec 18 IOPS
osd.12: bench: wrote 1 GiB in blocks of 4 MiB in 13.235 sec at 77 MiB/sec 19 IOPS
osd.13: bench: wrote 1 GiB in blocks of 4 MiB in 10.4559 sec at 98 MiB/sec 24 IOPS
osd.14: bench: wrote 1 GiB in blocks of 4 MiB in 12.469 sec at 82 MiB/sec 20 IOPS
osd.15: bench: wrote 1 GiB in blocks of 4 MiB in 17.434 sec at 59 MiB/sec 14 IOPS
osd.16: bench: wrote 1 GiB in blocks of 4 MiB in 11.7184 sec at 87 MiB/sec 21 IOPS
osd.17: bench: wrote 1 GiB in blocks of 4 MiB in 12.8702 sec at 80 MiB/sec 19 IOPS
osd.18: bench: wrote 1 GiB in blocks of 4 MiB in 20.1894 sec at 51 MiB/sec 12 IOPS
osd.19: bench: wrote 1 GiB in blocks of 4 MiB in 9.60049 sec at 107 MiB/sec 26 IOPS
osd.20: bench: wrote 1 GiB in blocks of 4 MiB in 15.0613 sec at 68 MiB/sec 16 IOPS
osd.21: bench: wrote 1 GiB in blocks of 4 MiB in 17.6074 sec at 58 MiB/sec 14 IOPS
osd.22: bench: wrote 1 GiB in blocks of 4 MiB in 16.39 sec at 62 MiB/sec 15 IOPS
osd.23: bench: wrote 1 GiB in blocks of 4 MiB in 15.2747 sec at 67 MiB/sec 16 IOPS
osd.24: bench: wrote 1 GiB in blocks of 4 MiB in 10.2462 sec at 100 MiB/sec 24 IOPS
osd.25: bench: wrote 1 GiB in blocks of 4 MiB in 13.5297 sec at 76 MiB/sec 18 IOPS
osd.26: bench: wrote 1 GiB in blocks of 4 MiB in 7.46824 sec at 137 MiB/sec 34 IOPS
osd.27: bench: wrote 1 GiB in blocks of 4 MiB in 11.2216 sec at 91 MiB/sec 22 IOPS
osd.28: bench: wrote 1 GiB in blocks of 4 MiB in 16.6205 sec at 62 MiB/sec 15 IOPS
osd.29: bench: wrote 1 GiB in blocks of 4 MiB in 10.1477 sec at 101 MiB/sec 25 IOPS
The different runs differ by ±1 IOPS.
Why are the osds 1,2,4,5,9,19,26 faster than the others?
Restarting an osd did change the result.
Could someone give me hint where to look further to find the reason?
Thanks
Lars
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
