Hi.
So, I have a test cluster made up of ludicrously overpowered machines with nothing but SSDs in them. Bonded 10Gbps NICs (802.3ad layer 2+3 xmit hash policy, confirmed ~19.8 Gbps throughput with 32+ threads). I'm running rados bench, and I am currently getting less than 1 MBps throughput:
sudo rados -N `hostname` bench 600 write -b 4096 -p volumes --no-cleanup -t 32 > bench_write_4096_volumes_1_32.out 2>&1'
Colocated journals on the same disk, so I'm not expecting optimum throughput, but previous tests on spinning disks have shown reasonable speeds (23MB/s, 4000-6000 iops) as opposed to the 150-450 iops I'm currently getting.
ceph_deploy@ssd-1001:~$ sudo ceph -s
cluster 4167d5f2-2b9e-4bde-a653-f24af68a45f8
health HEALTH_WARN clock skew detected on mon.ssd-1003
monmap e1: 3 mons at {ssd-1001=10.20.69.101:6789/0,ssd-1002=10.20.69.102:6789/0,ssd-1003=10.20.69.103:6789/0}, election epoch 20, quorum 0,1,2 ssd-1001,ssd-1002,ssd-1003
osdmap e344: 33 osds: 33 up, 33 in
pgmap v10600: 1650 pgs, 6 pools, 289 MB data, 74029 objects
466 GB used, 17621 GB / 18088 GB avail
1650 active+clean
client io 1263 kB/s wr, 315 op/s
ceph_deploy@ssd-1001:~$ sudo ceph osd tree
# id weight type name up/down reweight
-1 30.03 root default
-2 10.01 host ssd-1001
0 0.91 osd.0 up 1
1 0.91 osd.1 up 1
2 0.91 osd.2 up 1
3 0.91 osd.3 up 1
4 0.91 osd.4 up 1
5 0.91 osd.5 up 1
6 0.91 osd.6 up 1
7 0.91 osd.7 up 1
8 0.91 osd.8 up 1
9 0.91 osd.9 up 1
10 0.91 osd.10 up 1
-3 10.01 host ssd-1002
11 0.91 osd.11 up 1
12 0.91 osd.12 up 1
13 0.91 osd.13 up 1
14 0.91 osd.14 up 1
15 0.91 osd.15 up 1
16 0.91 osd.16 up 1
17 0.91 osd.17 up 1
18 0.91 osd.18 up 1
19 0.91 osd.19 up 1
20 0.91 osd.20 up 1
21 0.91 osd.21 up 1
-4 10.01 host ssd-1003
22 0.91 osd.22 up 1
23 0.91 osd.23 up 1
24 0.91 osd.24 up 1
25 0.91 osd.25 up 1
26 0.91 osd.26 up 1
27 0.91 osd.27 up 1
28 0.91 osd.28 up 1
29 0.91 osd.29 up 1
30 0.91 osd.30 up 1
31 0.91 osd.31 up 1
32 0.91 osd.32 up 1
The clock skew error can safely be ignored. It's something like 2-3 ms skew, I just haven't bothered configuring away the warning.
This is with a newly-created pool after deleting the last pool used for testing.
Any suggestions on where to start debugging?
thanks.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
