On Tue, Apr 27, 2010 at 11:41 PM, Tracy Reed <treed@xxxxxxxxxxxxxxx> wrote: > > Anyone know why my mirror devices would be doing (apparently) > unaligned writes to my ethernet SAN causing horrible performance and > massive seeking and lots of reading? BUT writing to the device > directly is very fast and no extra reads? I am measuring the > reads/writes on the SAN device in iostat as it is a Linux box. > > I am running the 2.6.18-164.11.1.el5xen xen/kernel which came with > CentOS 5.4 > > After spending a lot of time banging my head on this I seem to have > finally tracked it down to mirroring. I never would have thought it > would be this but it is extremely reproduceable. We're talking a > difference of 4-5x in write speed. Reads are equally fast everywhere. > > I am using AoE v72 kernel module (initiator) on a Dell R610's to talk > to vblade-19 (target) on Dell R710's all running CentOS 5.4. I have > striped two 7200 RPM SATA disks and exported the md with AoE (although > I have done these tests with individual disks also). Read performance > from a raw device is excellent: > > # dd of=/dev/null if=/dev/xvdg1 bs=4096 count=3000000 > 3000000+0 records in > 3000000+0 records out > 12288000000 bytes (12 GB) copied, 106.749 seconds, 115 MB/s > > or from a mirror: > > # dd if=foo of=/dev/null bs=4096 > 1073916+0 records in > 1073916+0 records out > 4398759936 bytes (4.4 GB) copied, 37.7441 seconds, 117 MB/s > > foo is a 4.4G file I created in the filesystem. > > I always dropped the cache with: > > echo 1 > /proc/sys/vm/drop_caches > > on both target and initiator before starting the test. This is great > for just a single gig-e link. This suggests that the network/SAN is > fine. > > And iostat shows only writes and no reads happening. > > However, write performance to a mirror is odious. Typically around > 20MB/s. > > # dd if=/dev/zero of=foo bs=4096 count=3000000 > 1724073+0 records in > 1724073+0 records out > 7061803008 bytes (7.1 GB) copied, 324.606 seconds, 21.8 MB/s > > It should be more like 70MB/s per disk or better (7200rpm SATA) and > max out my gig-e with write performance similar to the above read > performance. I mentioned above that I suspect these are somewhow > unaligned writes because when running iostat on the target machine I > can see lots of reads happening which are surely causing seeks and > killing performance. Typical is something like 8MB/s of reads while > doing 16MB/s of writes. > > I have tried manually aligning the disk by setting the beginning of > data on the partition from 63 to 64 (although I don't think this > should matter for a mirror as much as a stripe or raid5, right?) and I have > tried changing the disk geometry to account for the extra partition > table which causes a half-block page-cache misalignment as described > by the ever insightful Kelsey Hudson in his writeup on the issue here: > > http://copilotco.com/Virtualization/wiki/aoe-caching-alignment.pdf/at_download/file > > All to no avail. It remains that whenever I write to the mirrored > disks performance is terrible but when I write to each individual > block device with a filesystem on it performance is good. It seems to > point to some sort of problem with the mirroring. > > Any ideas or suggestions would be very appreciated very much. > > -- > Tracy Reed > http://tracyreed.org > Can you repeat the test using your network, but not using the mirror just to absolutely confirm that it's fine without the mirror? Then can you test with one disk in the 'mirror' and the other missing? Finally can you test it one last time after adding the other disk back to the mirror //AND// waiting for it to finish resync? -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
