>> I have sometimes vitnessed a RAID set slowing down tremendously and after some >> research, finding a single drive performing very badly (often down to 1-2% of >> what it should do). In zfs, I found this with zpool iostat (IIRC, that was some >> time back), but I'm not using zfs at home, just md. A friend just had a similar >> issue, so I tried nosing around looking for some counters to tell me what was >> lagging, but found none. Luckily, the raid only had six drives, so we tried >> hdparm -t on each of them, and one of them stood out with a speed of well below >> 1MB/s (the others were around 100MB/s, these being a diversity of old 1TB >> drives). Then I checked a drive that was kicked out of my home raid the other >> day, apparently for no reason (smart data looking ok etc, same thing with my >> friend's disk) and same thing there - perhaps 2MB/s on a Western Digital RE4 (6 >> years spinning time in one hour at this moment), which should be something like >> 150MB/s or thereabout. > > If you catch this slowdown event while it is occuring, you can run > > iostat -x /dev/sd? 2 > > to get a summary of disk I/O stats, including "utilization" percentage. The > slow disks will display 100% in the rightmost column, while others will be > mostly idle. Well, the bad drives are replaced on both machines, so I can't do it realtime, but sar -d -f /var/log/sysstat/sa06 tells me how things looked during the monthly check 12:00:01 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:05:01 AM dev7-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:05:01 AM dev8-0 9.63 14.92 114.92 13.49 0.09 9.46 8.53 8.21 12:05:01 AM dev9-0 0.36 0.00 53.35 146.94 0.00 0.00 0.00 0.00 12:05:01 AM dev9-1 7.89 16.60 108.30 15.83 0.00 0.00 0.00 0.00 12:05:01 AM dev8-16 4.16 4873.93 21.27 1175.59 0.39 92.81 5.15 2.15 12:05:01 AM dev8-32 4.17 4877.82 20.84 1175.48 0.36 86.09 4.92 2.05 12:05:01 AM dev8-48 13.51 4876.09 139.18 371.28 0.49 36.26 5.43 7.33 12:05:01 AM dev8-64 6.25 4873.02 27.95 784.65 0.73 116.28 8.19 5.11 12:05:01 AM dev8-96 4.00 4813.88 19.48 1207.10 24.77 6305.24 249.40 99.86 12:05:01 AM dev8-80 2.04 0.59 11.88 6.11 0.02 7.46 6.57 1.34 12:05:01 AM dev8-112 4.15 4876.01 12.37 1176.78 0.38 92.59 5.13 2.13 12:05:01 AM dev9-2 1.45 1.31 9.99 7.78 0.00 0.00 0.00 0.00 12:05:01 AM dev252-0 1.30 1.31 9.99 8.70 0.02 16.26 9.76 1.27 12:05:01 AM dev252-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:05:01 AM dev252-2 0.29 0.00 53.35 182.00 0.95 3246.68 698.91 20.49 12:05:01 AM dev252-3 7.76 16.60 108.30 16.09 0.12 15.02 9.21 7.15 and dev8-96 matches sdg, which was the troublesome drive. I guess I'll have to write a zabbix plugin for this (after having checked that noone else has done so already) thanks roy -- Roy Sigurd Karlsbakk (+47) 98013356 http://blogg.karlsbakk.net/ GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt -- Hið góða skaltu í stein höggva, hið illa í snjó rita.
