Hello everyone! I have built two clusters with old hardware, which is lying around, the possibility to upgrade is there. The clusters main usecase is hot backup. This means it's getting written 24/7 where 99% is writing and 1% is reading. It should be based on harddisks. At the moment, the nodes look like this: 8 Nodes Worst CPU: i7-3930K (up to i7-6850K) Worst ammount of RAM: 24GB (up to 64GB) HDD Layout: 1x 1TB 4x 2TB 1x 6TB all sata, some just 5400rpm I had to put the OS on the 6TB HDDs, because there are no more sata connections on the motherboard. The servers, which have to be backed up, have mounted the ceph with cephfs. 99% of the files, that have to be backed up, are harddisk images, so sizes from 5GB to 1TB. All files are written to an erasure-coded pool with k=6 m=2, compression is on passive snappy, default settings. I'm getting really bad performace with this setup. This is a bench, run with: "rados -p ec_test bench -b 524288 60 write" while normal operations: Total time run: 63.4957 Total writes made: 459 Write size: 524288 Object size: 524288 Bandwidth (MB/sec): 3.61442 Stddev Bandwidth: 3.30073 Max bandwidth (MB/sec): 16 Min bandwidth (MB/sec): 0 Average IOPS: 7 Stddev IOPS: 6.6061 Max IOPS: 32 Min IOPS: 0 Average Latency(s): 2.151 Stddev Latency(s): 2.3661 Max latency(s): 14.0916 Min latency(s): 0.0420954 Cleaning up (deleting benchmark objects) Removed 459 objects Clean up completed and total clean up time :35.6908 [root@testnode01 ~]# ceph osd perf osd commit_latency(ms) apply_latency(ms) 6 655 655 9 13 13 11 15 15 7 17 17 10 19 19 8 12 12 24 153 153 25 22 22 47 20 20 46 23 23 45 43 43 44 8 8 16 26 26 15 18 18 14 14 14 13 23 23 12 47 47 18 595 595 1 20 20 38 25 25 17 17 17 0 317 317 37 19 19 19 14 14 2 16 16 39 9 9 20 16 16 3 18 18 40 10 10 21 23 23 4 17 17 41 29 29 5 18 18 42 16 16 22 16 16 23 13 13 26 20 20 27 10 10 28 28 28 29 13 13 30 34 34 31 10 10 32 31 31 33 44 44 34 21 21 35 22 22 36 295 295 43 9 9 What do you think is the most obvious Problem? - The one 6TB disk, per node? - The OS on the 6TB disk? What would you suggest? What I hope to replace with this setup: 6 servers, each with 4x3TB disks, with lvm, no redundancy. (two times, that's why I have set up two clusters) Thanks in advance Simon _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |