On Fri, Jan 13 2006, Matt Darcy wrote: > Matt Darcy wrote: > > > > >> > >> > >>I can now provide further updates for this, although this are not > >>really super useful. > >> > >>I've copied the linux-raid list in as well, as after a little more > >>testing on my part I'd appriciate some input from the raid guys also. > >> > >>First of all, please ignore the comments above, there was a problem > >>with grub and it actually "failed back" and booted into the older git > >>release, so my initial test was actually done running the wrong kenel > >>which I didn't notice. Appologies to all for this. > >> > >>Last nights tests where done using the correct kernel (I fixed the > >>grub typo) 2.6.15-g5367f2d6 > >> > >>The details I have are as follows. > >> > >>I can run the machine accessing the 7 maxtor SATA disks as individual > >>disks for around 12 hours now, without any hangs or errors or any > >>real problems. I've not hit them very hard, but initial performance > >>seems fine and more than usable. > >> > >>The actual problems occurr when including these disks in a raid group. > >> > >>root@berger:~# fdisk -l /dev/sdc > >> > >>Disk /dev/sdc: 251.0 GB, 251000193024 bytes > >>255 heads, 63 sectors/track, 30515 cylinders > >>Units = cylinders of 16065 * 512 = 8225280 bytes > >> > >> Device Boot Start End Blocks Id System > >>/dev/sdc1 1 30515 245111706 fd Linux raid > >>autodetect > >> > >>root@berger:~# fdisk -l /dev/sde > >> > >>Disk /dev/sde: 251.0 GB, 251000193024 bytes > >>255 heads, 63 sectors/track, 30515 cylinders > >>Units = cylinders of 16065 * 512 = 8225280 bytes > >> > >> Device Boot Start End Blocks Id System > >>/dev/sde1 1 30515 245111706 fd Linux raid > >>autodetect > >> > >> > >>As you can see from my two random disks examples, they are > >>partitioned and makred as raid auto detect. > >> > >>I issue the mdadm command to build the raid 5 array > >> > >>mdadm -C /dev/md6 -l5 -n6 -x1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 > >>/dev/sdg1 /dev/sdh1 /dev/sdi1 > >> > >>and the array starts to build....... > >> > >>md6 : active raid5 sdh1[7] sdi1[6](S) sdg1[4] sdf1[3] sde1[2] sdd1[1] > >>sdc1[0] > >> 1225558080 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] > >> [>....................] recovery = 0.1% (374272/245111616) > >>finish=337.8min speed=12073K/sec > >> > >> > >>however at around %25 - %40 completion the box will simpley just hang > >>- I'm getting no on screen messages and the sylog is not reporting > >>anything. > >> > >>SysRQ is unusable. > >> > >>I'm open to options on how to resolve this and move the driver > >>forward (assuming it is the drivers interfaction with the raid sub > >>system) > >>or > >>how to get some meaningful debug out to report back to the > >>appropriate development groups. > >> > >>thanks. > >> > >>Matt. > >> > >> > >> > >Further further information > > > >The speed that the raid array is being built att appears to drop as > >the array is created > > > >[=====>...............] recovery = 29.2% (71633360/245111616) > >finish=235.1min speed=12296K/sec > >[=====>...............] recovery = 29.3% (71874512/245111616) > >finish=235.2min speed=12269K/sec > >[=====>...............] recovery = 29.4% (72115872/245111616) > >finish=236.0min speed=12209K/sec > >[=====>...............] recovery = 29.7% (72839648/245111616) > >finish=237.4min speed=12091K/sec > >[=====>...............] recovery = 29.8% (73078560/245111616) > >finish=238.6min speed=12010K/sec > >[=====>...............] recovery = 29.8% (73139424/245111616) > >finish=350.5min speed=8176K/sec > >[=====>...............] recovery = 29.8% (73139424/245111616) > >finish=499.6min speed=5735K/sec > >[=====>...............] recovery = 29.8% (73139776/245111616) > >finish=691.0min speed=4147K/sec > > > >Now the box is hung > > > >I didn't notice this until about %20 through the creation of the array > >then I started paying attention to this. These snap shots are taken > >every 30 seconds > > > >So the problem appears to sap bandwidth on the card to the point there > >the box hangs. > > > >This may have some relevance, or it may not, but worth mentioning at > >least. > > > >Matt > > > > > > > First - a quick response to John Stoffels comments. > > Both disks and controller on the latest Bios/firmware versions (thanks > for making me point this out) > > I created a much smaller array (3 disks 1 spare) today and again around > %35 through the creation of the array the whole machine hung, no warning > no errors no logging. > The speed parameter from /proc/mdstat stayed constant to around %30 > (which explained why I perhaps didn't notice this earlier) and like the > creation of the large raid 5 array took a massive nose dive in speed > over about 180 seconds to the point where the box hung. > > Its almost as if there is an "IO leak" which is the only way I can think > of to describe it.the card / system performaces quite well as individual > disks, but as soon as its entered into a raid 5 configuration using the > any number of disks the creation of the array appears to be fine until > around %20-%30 through the assembly, the speed of the arrays creations > plummits and the machine hangs. > > I'm not too sure how to take this further as I get no warnings (other > than the arrays creation time slowing) - I can't use any tools like > netdump or sysRQ. > > I'll try some additional raid tests (such as raid0 or raid1 across more > disks) to see how that works. But as it stands I'm not sure how to get > additional information. You could try and monitor /proc/meminfo and /proc/slabinfo as the system begins to slow to a crawl, any leaks of io structures should be visible there as well. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
