Mr. Jennings, The first thing I would try is to run iostat with -x so that you get per disk details stats. In particular, you want to see if you have 1 disk that has a much higher utilization than the others. Raid tends to run at the speed of the slowest drive. If you do see a slow drive, it might mean that the drive is bad, or it might just mean that the one drive has write caching disabled. Your write operations are small (512 bytes) and your array is doing a lot of read/modify/write raid updates, so the raid array is not taking much advantage of your linear writes. I would also try writes that are much larger as another data point. If you can take the array off-line, you might want to write test to each drive. You can dd read a segment to a temp file and then dd write it back out so you are not actually destroying any data. Doug Dumitru EasyCo LLC On Wed, Jan 25, 2012 at 10:15 AM, Seth Jennings <spartacus06@xxxxxxxxx> wrote: > The write performance on my raid5 took a nose dive yesterday and I can't > figure out what is to blame. iostat is showing 98% iowait with 2-5 tps per > array disk (?!). I'm including as much information as I can think to include > without overwhelming anyone inclined to help me debug this. > > Also, I'm familiar with kernel internals/debugging so just let me know if you > need more information. > > Thanks > -- > Seth > > ====================== > All disks are SATA and pass SMART health assessment. No errors in dmesg. > > Setup: > /dev/sda: 250GB single part > /dev/sdb: 250GB single part > /dev/sdc: 500GB, 2x250GB parts > /dev/sdd: 320GB, 250GB and 70GB parts > > /dev/sda: > Timing cached reads: 2294 MB in 2.00 seconds = 1147.32 MB/sec > Timing buffered disk reads: 186 MB in 3.02 seconds = 61.69 MB/sec > sjennings@cerebrum:~$ sudo hdparm -Tt /dev/sdb > > /dev/sdb: > Timing cached reads: 2250 MB in 2.00 seconds = 1125.59 MB/sec > Timing buffered disk reads: 184 MB in 3.01 seconds = 61.05 MB/sec > > /dev/sdc: > Timing cached reads: 2172 MB in 2.00 seconds = 1086.00 MB/sec > Timing buffered disk reads: 392 MB in 3.01 seconds = 130.36 MB/sec > > /dev/sdd: > Timing cached reads: 2220 MB in 2.00 seconds = 1110.60 MB/sec > Timing buffered disk reads: 236 MB in 3.02 seconds = 78.15 MB/sec > > /dev/md0: > Version : 0.90 > Creation Time : Mon Jul 12 08:32:58 2010 > Raid Level : raid5 > Array Size : 732587712 (698.65 GiB 750.17 GB) > Used Dev Size : 244195904 (232.88 GiB 250.06 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 0 > Persistence : Superblock is persistent > > Update Time : Wed Jan 25 09:51:25 2012 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 64K > > UUID : cf70d928:8ad26aac:17383c13:03badee3 > Events : 0.2068 > > Number Major Minor RaidDevice State > 0 8 1 0 active sync /dev/sda1 > 1 8 17 1 active sync /dev/sdb1 > 2 8 34 2 active sync /dev/sdc2 > 3 8 49 3 active sync /dev/sdd1 > > > --- Physical volume --- > PV Name /dev/md0 > VG Name raid5vg > PV Size 698.65 GiB / not usable 1.44 MiB > Allocatable yes > PE Size 4.00 MiB > Total PE 178854 > Free PE 57254 > Allocated PE 121600 > PV UUID F038RQ-reR4-BSPy-43lA-UJI4-uoMY-XQk23n > > --- Volume group --- > VG Name raid5vg > System ID > Format lvm2 > Metadata Areas 1 > Metadata Sequence No 47 > VG Access read/write > VG Status resizable > MAX LV 0 > Cur LV 6 > Open LV 3 > Max PV 0 > Cur PV 1 > Act PV 1 > VG Size 698.65 GiB > PE Size 4.00 MiB > Total PE 178854 > Alloc PE / Size 121600 / 475.00 GiB > Free PE / Size 57254 / 223.65 GiB > VG UUID 9KjnCN-l4gT-jUkR-gqt5-DyDR-GeGX-20DmJc > > --- Logical volume --- > LV Name /dev/raid5vg/home > VG Name raid5vg > LV UUID flP8gL-adJq-Ur0d-Nsl0-olZ8-tzpi-fjqGi6 > LV Write Access read/write > LV Status available > # open 1 > LV Size 250.00 GiB > Current LE 64000 > Segments 2 > Allocation inherit > Read ahead sectors auto > - currently set to 768 > Block device 252:0 > > /dev/mapper/raid5vg-home is mounted at /home type ext4 (rw,noatime) > > read /home (dm-0 on md0): > > dd if=ubuntu-11.04-alternate-i386.iso of=/dev/null (not in page cache) > 1419416+0 records in > 1419416+0 records out > 726740992 bytes (727 MB) copied, 5.65182 s, 129 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.99 0.00 27.36 21.39 0.00 48.26 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 531.00 37632.00 0.00 37632 0 > sdb 451.00 37648.00 0.00 37648 0 > sdc 532.00 37760.00 0.00 37760 0 > md0 2751.00 150912.00 0.00 150912 0 > sdd 453.00 37712.00 0.00 37712 0 > dm-0 2751.00 150912.00 0.00 150912 0 > > so reading is good. > > write /home: > > dd if=/dev/zero of=zeroes > <ctrl-c> > 208385+0 records in > 208384+0 records out > 106692608 bytes (107 MB) copied, 27.6739 s, 3.9 MB/s > > qavg-cpu: %user %nice %system %iowait %steal %idle > 0.52 0.00 1.55 97.94 0.00 0.00 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 2.00 192.00 444.00 192 444 > sdb 5.00 192.00 1020.00 192 1020 > sdc 3.00 192.00 444.00 192 444 > md0 34.00 768.00 1344.00 768 1344 > sdd 5.00 192.00 1020.00 192 1020 > dm-0 34.00 768.00 1344.00 768 1344 > > so writing is aweful (2-5 tps per disk with 98% iowait?!). > > write /dev/sdc1 (non-raid part in /dev/sdc) > > dd if=/dev/zero of=zeroes bs=4096 count=100000 > 100000+0 records in > 100000+0 records out > 409600000 bytes (410 MB) copied, 2.64131 s, 155 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 1.50 0.00 32.00 57.00 0.00 9.50 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 0.00 0.00 0.00 0 0 > sdb 0.00 0.00 0.00 0 0 > sdc 267.00 0.00 135680.00 0 135680 > md0 0.00 0.00 0.00 0 0 > sdd 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > > so writing to non-raid partition of one of the disks is good. > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Doug Dumitru EasyCo LLC -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
