On Mon, 2005-07-11 at 11:11 -0400, Dan Christensen wrote: > I was wondering what I should expect in terms of streaming read > performance when using (software) RAID-5 with four SATA drives. I > thought I would get a noticeable improvement compared to reads from a > single device, but that's not the case. I tested this by using dd to > read 300MB directly from disk partitions /dev/sda7, etc, and also using > dd to read 300MB directly from the raid device (/dev/md2 in this case). > I get around 57MB/s from each of the disk partitions that make up the > raid device, and about 58MB/s from the raid device. On the other > hand, if I run parallel reads from the component partitions, I get > 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s. > > Before each read, I try to clear the kernel's cache by reading > 900MB from an unrelated partition on the disk. (Is this guaranteed > to work? Is there a better and/or faster way to clear cache?) > > I have AAM quiet mode/low performance enabled on the drives, but (a) > this shouldn't matter too much for streaming reads, and (b) it's the > relative performance of the reads from the partitions and the RAID > device that I'm curious about. > > I also get poor write performance, but that's harder to isolate > because I have to go through the lvm and filesystem layers too. > > I also get poor performance from my RAID-1 array and my other > RAID-5 arrays. > > Details of my tests and set-up below. > > Thanks for any suggestions, > > Dan > > > System: > - Athlon 2500+ > - kernel 2.6.12.2 (also tried 2.6.11.11) > - four SATA drives (3 160G, 1 200G); Samsung Spinpoint > - SiI3114 controller (latency_timer=32 by default; tried 128 too) only 1 card? 4 port? try some other brand card and try to use several cards at the same time. i met some poor cards before. > - 1G ram > - blockdev --getra /dev/sda --> 256 (didn't play with these) > - blockdev --getra /dev/md2 --> 768 (didn't play with this) > - tried anticipatory, deadline and cfq schedules, with no significant > difference. > - machine essentially idle during tests > > Here is part of /proc/mdstat (the full output is below): > > md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0] > 218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU] > > Here's the test script and output: > > # Clear cache: > dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1 > for f in sda7 sdb5 sdc5 sdd5 ; do > echo $f > dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec > echo > done > > # Clear cache: > dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1 > for f in md2 ; do > echo $f > dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec > echo > done > > Output: > > sda7 > 314572800 bytes transferred in 5.401071 seconds (58242671 bytes/sec) > > sdb5 > 314572800 bytes transferred in 5.621170 seconds (55962158 bytes/sec) > > sdc5 > 314572800 bytes transferred in 5.635491 seconds (55819947 bytes/sec) > > sdd5 > 314572800 bytes transferred in 5.333374 seconds (58981951 bytes/sec) > > md2 > 314572800 bytes transferred in 5.386627 seconds (58398846 bytes/sec) > > # cat /proc/mdstat > md1 : active raid5 sdd1[2] sdc1[1] sda2[0] > 578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU] > > md4 : active raid5 sdd2[3] sdc2[2] sdb2[1] sda6[0] > 30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU] > > md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0] > 218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU] > > md3 : active raid5 sdd6[3] sdc6[2] sdb6[1] sda8[0] > 218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU] > > md0 : active raid1 sdb1[0] sda5[1] > 289024 blocks [2/2] [UU] > > # mdadm --detail /dev/md2 > /dev/md2: > Version : 00.90.01 > Creation Time : Mon Jul 4 23:54:34 2005 > Raid Level : raid5 > Array Size : 218612160 (208.48 GiB 223.86 GB) > Device Size : 72870720 (69.49 GiB 74.62 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 2 > Persistence : Superblock is persistent > > Update Time : Thu Jul 7 21:52:50 2005 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 64K > > UUID : c4056d19:7b4bb550:44925b88:91d5bc8a > Events : 0.10873823 > > Number Major Minor RaidDevice State > 0 8 7 0 active sync /dev/sda7 > 1 8 21 1 active sync /dev/sdb5 > 2 8 37 2 active sync /dev/sdc5 > 3 8 53 3 active sync /dev/sdd5 > > - > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
