I can only comment I the log. I would recommend using three logs (6 disks as mirror pairs) per system, and add a crush map hierarchy level for cache drive so that any given pg will never mirror twice to the same log. That'll also reduce your failure domain.
On Jan 29, 2014 4:26 PM, "Geraint Jones" <geraint@xxxxxxxxxx> wrote:
Hi GuysWe have the current config :2 x Storage servers, 128gb RAM, dual E5-2609, LSI MegaRAID SAS 9271-4i, each server has 24 x 3tb Disks – these were originally set as 8 groups of 3 Disk RAID0 (we are slowly moving to one OSD one disk) We initially had the journals stored on an SSD, however after a disk failure this lead to terrible performance (a_wait on the SSD was huge) so we added some PCIe SSDs, this didn’t change a lot and we had the big a_wait on the SSDs still. So we removed the SSDs. This made recovery pretty good (we were able to carry on working while recovery was taking place, previously recovery = VM’s down).Now two mornings in a row I have been paged due to really slow I/O.It appears that the background deep scrubbing was using all the IO the disks had, this is the output of iostat –xk 3 during scrubbing:avg-cpu: %user %nice %system %iowait %steal %idle31.25 0.00 6.17 10.79 0.00 51.78Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sda 0.00 12.33 0.00 26.67 0.00 130.67 9.80 0.07 2.55 0.00 2.55 2.55 6.80sdi 0.00 8.00 269.33 319.33 31660.00 5124.33 124.98 4.51 7.47 5.68 8.98 1.67 98.13sdk 0.00 0.00 9.00 88.00 117.33 2208.17 47.95 0.10 1.03 10.67 0.05 0.95 9.20sdj 0.00 0.00 5.33 114.67 26.67 928.50 15.92 0.09 0.77 12.75 0.21 0.59 7.07sdg 0.33 0.00 124.33 39.00 15537.33 488.83 196.24 0.46 2.69 3.53 0.00 2.05 33.47sdf 0.00 10.67 12.33 277.33 96.00 2481.83 17.80 0.88 3.05 14.27 2.55 0.63 18.27sde 0.00 0.00 3.00 32.67 381.33 280.33 37.10 0.10 2.84 7.11 2.45 1.68 6.00sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdc 0.00 0.00 0.00 28.00 0.00 179.00 12.79 0.05 1.90 0.00 1.90 0.76 2.13sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00dm-0 0.00 0.00 0.00 32.33 0.00 130.67 8.08 0.07 2.14 0.00 2.14 2.10 6.80dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdp 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00avg-cpu: %user %nice %system %iowait %steal %idle7.10 0.00 6.08 23.03 0.00 63.79Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sda 0.00 0.00 0.00 0.33 0.00 1.33 8.00 0.00 0.00 0.00 0.00 0.00 0.00sdc 0.00 0.00 0.00 13.00 0.00 168.00 25.85 0.01 0.92 0.00 0.92 0.92 1.20sdd 0.00 0.00 21.67 142.33 181.33 24873.33 305.54 29.24 175.61 76.62 190.68 6.09 99.87sde 0.33 0.00 273.33 10.00 32257.33 242.67 229.41 1.45 5.11 5.25 1.20 3.13 88.67sdf 0.00 0.00 0.67 44.00 2.67 520.00 23.40 0.03 0.75 20.00 0.45 0.75 3.33sdg 0.00 0.00 2.00 13.33 68.00 170.67 31.13 0.24 13.30 70.00 4.80 12.61 19.33sdi 0.00 0.00 1.00 47.67 5.33 767.83 31.77 0.08 1.62 8.00 1.48 0.30 1.47sdh 0.00 0.00 1.00 21.00 54.67 264.00 28.97 0.06 1.94 10.67 1.52 0.61 1.33sdj 1.33 0.00 344.33 16.67 42037.33 212.00 234.07 1.50 4.18 4.32 1.28 2.66 95.87sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00dm-0 0.00 0.00 0.00 0.33 0.00 1.33 8.00 0.00 0.00 0.00 0.00 0.00 0.00dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00I have disabled scrubbing to return systems to a usable state (osd set noscrub and osd set no deep-scrub)And here is how our iostats look now.avg-cpu: %user %nice %system %iowait %steal %idle33.33 0.00 6.73 0.67 0.00 59.27
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sda 0.00 6.67 0.00 20.00 0.00 88.00 8.80 0.05 2.53 0.00 2.53 2.53 5.07sdi 0.00 0.00 1.67 89.33 9.33 520.00 11.63 0.03 0.34 15.20 0.06 0.34 3.07sdk 0.00 0.00 1.33 139.33 8.00 1203.67 17.23 0.08 0.56 13.00 0.44 0.15 2.13sdj 0.00 0.67 2.67 175.00 126.67 4583.33 53.02 0.02 0.12 4.00 0.06 0.11 1.87sdg 0.00 0.00 1.00 151.00 5.33 1288.17 17.02 0.03 0.23 13.33 0.14 0.17 2.53sdf 0.00 0.00 3.33 93.00 20.00 870.33 18.48 0.03 0.36 9.20 0.04 0.33 3.20sde 0.00 0.00 0.00 13.33 0.00 72.00 10.80 0.00 0.20 0.00 0.20 0.20 0.27sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdc 0.00 0.00 1.33 26.00 8.00 182.67 13.95 0.02 0.68 9.00 0.26 0.63 1.73sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00dm-0 0.00 0.00 0.00 21.67 0.00 88.00 8.12 0.05 2.34 0.00 2.34 2.34 5.07dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdp 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00avg-cpu: %user %nice %system %iowait %steal %idle8.40 0.00 3.65 4.41 0.00 83.55Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdc 0.00 0.00 0.00 62.67 0.00 778.00 24.83 0.18 2.89 0.00 2.89 0.28 1.73sdd 0.00 10.33 1.67 160.00 13.33 1163.33 14.56 23.88 41.49 22.40 41.69 2.41 38.93sde 0.00 0.00 0.00 50.33 0.00 601.17 23.89 0.11 2.09 0.00 2.09 0.21 1.07sdf 0.00 0.00 0.00 106.67 0.00 1260.67 23.64 0.41 3.81 0.00 3.81 0.27 2.93sdg 0.00 0.00 0.00 71.33 0.00 863.50 24.21 0.15 2.15 0.00 2.15 0.37 2.67sdi 0.00 0.00 0.00 23.67 0.00 366.67 30.99 0.01 0.45 0.00 0.45 0.45 1.07sdh 0.00 0.00 0.00 73.00 0.00 910.67 24.95 0.20 2.74 0.00 2.74 0.24 1.73sdj 0.00 0.00 0.00 82.67 0.00 899.67 21.77 0.28 3.39 0.00 3.39 0.23 1.87sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00Now clearly we can’t live with scrubbing off for very long, so what can I do to stop scrubbing blocking IO or if this was you how would you reconfigure/rearchitect things ?As an aside has anyone used one of these to hold the journals and would 1 be enough for 24 journals ? : http://www.amazon.com/FUSiON-iO-420GB-Solid-State-Drive/dp/B00DVMPXV0/ref=sr_1_1?ie=UTF8&qid=1391039407&sr=8-1&keywords=fusion-ioThanks!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
