Hi,
are any of those old disks SMR ones? Because they will absolutely
destroy any kind of performance (ceph does not use writecaches due to
powerloss concerns, so they kinda do their whole magic for each
writerequest).
Greetings
On 9/6/21 10:47 AM, Simon Sutter wrote:
Hello everyone!
I have built two clusters with old hardware, which is lying around, the possibility to upgrade is there.
The clusters main usecase is hot backup. This means it's getting written 24/7 where 99% is writing and 1% is reading.
It should be based on harddisks.
At the moment, the nodes look like this:
8 Nodes
Worst CPU: i7-3930K (up to i7-6850K)
Worst ammount of RAM: 24GB (up to 64GB)
HDD Layout:
1x 1TB
4x 2TB
1x 6TB
all sata, some just 5400rpm
I had to put the OS on the 6TB HDDs, because there are no more sata connections on the motherboard.
The servers, which have to be backed up, have mounted the ceph with cephfs.
99% of the files, that have to be backed up, are harddisk images, so sizes from 5GB to 1TB.
All files are written to an erasure-coded pool with k=6 m=2, compression is on passive snappy, default settings.
I'm getting really bad performace with this setup.
This is a bench, run with: "rados -p ec_test bench -b 524288 60 write" while normal operations:
Total time run: 63.4957
Total writes made: 459
Write size: 524288
Object size: 524288
Bandwidth (MB/sec): 3.61442
Stddev Bandwidth: 3.30073
Max bandwidth (MB/sec): 16
Min bandwidth (MB/sec): 0
Average IOPS: 7
Stddev IOPS: 6.6061
Max IOPS: 32
Min IOPS: 0
Average Latency(s): 2.151
Stddev Latency(s): 2.3661
Max latency(s): 14.0916
Min latency(s): 0.0420954
Cleaning up (deleting benchmark objects)
Removed 459 objects
Clean up completed and total clean up time :35.6908
[root@testnode01 ~]# ceph osd perf
osd commit_latency(ms) apply_latency(ms)
6 655 655
9 13 13
11 15 15
7 17 17
10 19 19
8 12 12
24 153 153
25 22 22
47 20 20
46 23 23
45 43 43
44 8 8
16 26 26
15 18 18
14 14 14
13 23 23
12 47 47
18 595 595
1 20 20
38 25 25
17 17 17
0 317 317
37 19 19
19 14 14
2 16 16
39 9 9
20 16 16
3 18 18
40 10 10
21 23 23
4 17 17
41 29 29
5 18 18
42 16 16
22 16 16
23 13 13
26 20 20
27 10 10
28 28 28
29 13 13
30 34 34
31 10 10
32 31 31
33 44 44
34 21 21
35 22 22
36 295 295
43 9 9
What do you think is the most obvious Problem?
- The one 6TB disk, per node?
- The OS on the 6TB disk?
What would you suggest?
What I hope to replace with this setup:
6 servers, each with 4x3TB disks, with lvm, no redundancy. (two times, that's why I have set up two clusters)
Thanks in advance
Simon
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
