Hello, On Tue, 6 Sep 2016 12:57:55 +0200 lists wrote: > Hi all, > > We're pretty new to ceph, but loving it so far. > > We have a three-node cluster, four 4TB OSDs per node, journal (5GB) on > SSD, 10G ethernet cluster network, 64GB ram on the nodes, total 12 OSDs. > What SSD model (be precise)? Only one SSD? > We noticed the following output when using ceph bench: > > > root@ceph1:~# rados bench -p scbench 600 write --no-cleanup The end result of something shorter (like 60 seconds) would have been sufficient, too. And still be of interest. > > Maintaining 16 concurrent writes of 4194304 bytes for up to 600 seconds or 0 objects > > Object prefix: benchmark_data_pm1_36584 > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > > 0 0 0 0 0 0 - 0 > > 1 16 124 108 431.899 432 0.138315 0.139077 > > 2 16 237 221 441.928 452 0.169759 0.140138 > > 3 16 351 335 446.598 456 0.105837 0.139844 > > 4 16 466 450 449.938 460 0.140141 0.139716 > > 5 16 569 553 442.337 412 0.025245 0.139328 > > 6 16 634 618 411.943 260 0.0302609 0.147129 > > 7 16 692 676 386.233 232 1.01843 0.15158 > > 8 16 721 705 352.455 116 0.0224958 0.159924 > > 9 16 721 705 313.293 0 - 0.159924 > +------------------ notice the drop to zero for MB/s > > 10 16 764 748 299.163 86 0.0629263 0.20961 > > 11 16 869 853 310.144 420 0.0805086 0.204707 > > 12 16 986 970 323.295 468 0.175718 0.196822 > > 13 16 1100 1084 333.5 456 0.171172 0.19105 > > 14 16 1153 1137 324.819 212 0.0468416 0.188643 > > 15 16 1225 1209 322.363 288 0.0421159 0.195791 > > 16 16 1236 1220 304.964 44 1.28629 0.195499 > > 17 16 1236 1220 287.025 0 - 0.195499 > > 18 16 1236 1220 271.079 0 - 0.195499 > +------------------ notice again the drop to zero for MB/s > > 19 16 1324 1308 275.336 117.333 0.148679 0.231708 > > 20 16 1436 1420 283.967 448 0.120878 0.224367 > > 21 16 1552 1536 292.538 464 0.173587 0.218141 > > 22 16 1662 1646 299.238 440 0.141544 0.212946 > > 23 16 1720 1704 296.314 232 0.0273257 0.211416 > > 24 16 1729 1713 285.467 36 0.0215821 0.211308 > > 25 16 1729 1713 274.048 0 - 0.211308 > > 26 16 1729 1713 263.508 0 - 0.211308 > +------------------ notice again the drop to zero for MB/s > > 27 16 1787 1771 262.34 77.3333 0.0338129 0.241103 > > 28 16 1836 1820 259.97 196 0.183042 0.245665 > > 29 16 1949 1933 266.59 452 0.129397 0.239445 > > 30 16 2058 2042 272.235 436 0.165108 0.234447 > > 31 16 2159 2143 276.484 404 0.0466259 0.229704 > > 32 16 2189 2173 271.594 120 0.0206958 0.231772 > > So regular intervals, the "cur MB/s" appears to drop to zero. If > meanwhile we ALSO run iperf between two nodes, we can tell that the > network is fuctioning perfectly: while ceph bench goes to zero, iperf > continues at max speed. (10G ethernet) > No surprise there. > So it seems there is something slowing down ceph at 'regular' intervals. > Is this normal, and expected, or not? In which case: What do we need to > look at? > The most obvious and slowest thing in your setup, your storage. > During the 0 MB/sec, there is NO increased cpu usage: it is usually > around 15 - 20% for the four ceph-osd processes. > Watch your node(s) with atop or iostat. > Do we have an issue..? And if yes: Anyone with a suggestions where to > look at? > You will find that either your journal SSD is overwhelmed and a single SSD peaking around 500MB/s wouldn't be that surprising. Or that your HDDs can't scribble away at more than the speed above, the more likely reason. Even a combination of both. Ceph needs to flush data to the OSDs eventually (and that is usually more or less immediately with default parameters), so for a sustained, sequential write test you're looking at the speed of your HDDs. And that will be spiky of sorts, due to FS journals, seeks for other writes (replicas), etc. Christian > Some more details: > - ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432) > - Linux ceph2 4.4.15-1-pve #1 SMP Thu Jul 28 10:54:13 CEST 2016 x86_64 > GNU/Linux > > Thanks in advance, and best regards from the netherlands, > > MJ > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |