Hello list,
Just curious if anyone has ever seen this behavior and might have some ideas on how to troubleshoot it.
We're seeing very high iowait in iostat across all OSD's in on a single OSD host. It's very spiky - dropping to zero and then shooting up to as high as 400 in some cases. Despite this it does not seem to be having a major impact on the cluster performance as a whole.
Some more details:
3x OSD Nodes - Dell R730's: 24 cores @2.6GHz, 256GB RAM, 20x 1.2TB 10K SAS OSD's per node.
We're running ceph hammer.
Here's the output of iostat. Note that this is from a period when the cluster is not very busy but you can still see high spikes on a few OSD's. It's much worse during high load.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.50 0.00 6.00 24.00 0.00 8.00 0.00 8.00 8.00 0.40
sdb 0.00 0.00 0.00 60.00 0.00 808.00 26.93 0.00 0.07 0.00 0.07 0.03 0.20
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 67.00 0.00 1010.00 30.15 0.01 0.09 0.00 0.09 0.09 0.60
sde 0.00 0.00 0.00 93.00 0.00 868.00 18.67 0.00 0.04 0.00 0.04 0.04 0.40
sdf 0.00 0.00 0.00 57.50 0.00 572.00 19.90 0.00 0.03 0.00 0.03 0.03 0.20
sdg 0.00 1.00 0.00 3.50 0.00 22.00 12.57 0.75 16.00 0.00 16.00 2.86 1.00
sdh 0.00 0.00 1.50 25.50 6.00 458.50 34.41 2.03 75.26 0.00 79.69 3.04 8.20
sdi 0.00 0.00 0.00 30.50 0.00 384.50 25.21 2.36 77.51 0.00 77.51 3.28 10.00
sdj 0.00 1.00 1.50 105.00 6.00 925.75 17.50 10.85 101.84 8.00 103.18 2.35 25.00
sdl 0.00 0.00 2.00 0.00 320.00 0.00 320.00 0.01 3.00 3.00 0.00 2.00 0.40
sdk 0.00 1.00 0.00 55.00 0.00 334.50 12.16 7.92 136.91 0.00 136.91 2.51 13.80
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdo 0.00 0.00 1.00 0.00 4.00 0.00 8.00 0.00 4.00 4.00 0.00 4.00 0.40
sdp 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdq 0.50 0.00 756.00 0.00 93288.00 0.00 246.79 1.47 1.95 1.95 0.00 1.17 88.60
sdr 0.00 0.00 1.00 0.00 4.00 0.00 8.00 0.00 4.00 4.00 0.00 4.00 0.40
sds 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdt 0.00 0.00 0.00 36.50 0.00 643.50 35.26 3.49 95.73 0.00 95.73 2.63 9.60
sdu 0.00 0.00 0.00 21.00 0.00 323.25 30.79 0.78 37.24 0.00 37.24 2.95 6.20
sdv 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdw 0.00 0.00 0.00 31.00 0.00 689.50 44.48 2.48 80.06 0.00 80.06 3.29 10.20
sdx 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.50 0.00 6.00 24.00 0.00 8.00 0.00 8.00 8.00 0.40
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
