German Anders
Storage System Engineer Leader
Despegar | IT Team
office +54 11 4894 3500 x3408
mobile +54 911 3493 7262
mail ganders@xxxxxxxxxxxx
From the looks of it you are reaching the maxing out your OSD’s. Some of them are pushing over 500 iops, which is a lot for a 7.2k disk and at the high queue depths IO’s will have to wait a long time to reach the front of the queue. The only real thing I can suggest is to add more OSD’s which will spread your workload over more disks. It’s possible creating more PG’s may distribute the workload a little better, but I don’t see it making a massive difference.
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of German Anders
Sent: 10 June 2015 17:20
To: Ilya Dryomov
Cc: ceph-users; Nick Fisk
Subject: Re: krbd splitting large IO's into smaller IO's
hi guys, sorry that I hang on this email, I've four OSD servers with Ubuntu 14.04.1 LTS with 9 osd daemons each, 3TB drive size, and 3 ssd journal drives (each journal holds 3 osd daemons), the kernel version that I'm using is 3.18.3-031803-generic, and ceph version 0.82, I would like to know what would be the 'best' parameters in term of io for my 3TB devices, I've:
scheduler: deadline
max_hw_sectors_kb: 16383
max_sectors_kb: 4096
read_ahead_kb: 128
nr_requests: 128
I'm experience some high io waits on all the OSD servers:
avg-cpu: %user %nice %system %iowait %steal %idle
1.74 0.00 15.43 64.80 0.00 18.03
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1610.40 322.20 374.80 11.00 7940.80 1330.00 48.06 0.08 0.21 0.20 0.44 0.20 7.68
sdb 130.60 322.20 55.00 11.00 742.40 1330.00 62.80 0.02 0.23 0.17 0.51 0.19 1.28
md0 0.00 0.00 2170.80 332.40 8683.20 1329.60 8.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 2170.80 332.40 8683.20 1329.60 8.00 0.87 0.35 0.21 1.26 0.03 7.84
sdd 0.00 0.00 11.80 384.40 4217.60 33197.60 188.87 75.17 189.72 130.78 191.53 1.88 74.64
sdc 0.00 0.00 18.80 313.40 581.60 33154.40 203.11 78.09 235.08 66.85 245.17 2.16 71.84
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.80 78.20 181.40 10400.80 19204.80 228.09 31.75 110.93 43.09 140.18 2.99 77.52
sdg 0.00 0.00 1.60 304.60 51.20 31647.20 207.04 64.05 209.19 73.50 209.90 1.90 58.32
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 6.60 17.20 159.20 2784.80 247.39 0.27 9.14 12.12 8.00 3.19 7.60
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 13.40 120.00 428.80 8487.20 133.67 23.91 203.37 36.18 222.04 2.64 35.28
sdl 0.00 0.80 12.40 524.20 2088.80 40842.40 160.01 93.53 168.27 183.35 167.91 1.64 88.24
sdn 0.00 1.40 4.00 433.80 92.80 35926.40 164.55 88.72 196.29 299.40 195.33 1.71 74.96
sdm 0.00 0.00 0.60 544.60 19.20 40348.00 148.08 118.31 217.00 17.33 217.22 1.67 90.80Thanks in advance,
Best regards,
German Anders
Storage System Engineer Leader
Despegar | IT Team
office +54 11 4894 3500 x3408
mobile +54 911 3493 7262
mail ganders@xxxxxxxxxxxx
![]()
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
