I re-ran the test using buffered IO, with both XDD and FIO. Except that I was using XDD and FIO, the tests were essentially the same -- TC=4, QD=1, BS=1M, DIR=Write, Order=SEQ. I've switched to using spinning disks, since several people on this list indicated that using DDR Disk devices with it's ultra low latency could cause unintended behavior in RAID6. My theoretical MAX TP is 1080 - 1620 MB/s for 6 HDD's that each have a TP of about 270 MB/s on the outer edge. My SAS infrastructure is such that I shouldn't have a problem achieving the Theoretical MAX, when testing on the outer edge. With Buffered IO... FIO reported a sustained TP of 2200 MB/s. As this is well above the theoretical MAX, I didn't trust it. I ran the same test with XDD. It reported a TP of 570 MB/s. When I watched the system via vmstat and iostat while running both of these tests, both tools indicated that I was doing about 570 MB/s while running both XDD and FIO. So there must be something wrong in the way FIO is calculating TP when using buffered IO. When running both FIO and XDD using direct IO, I was able to get much better performance (about 1040 MB/s) with the above paramaters (TC=4, QD=1, BS=1M, DIR=Write, Order=SEQ). I am able to confirm that TP using vmstat and iostat. While running with both Buffered-IO and Direct-IO, I watched the IO's in both iostat and vmstat and can confirm that there weren't any reads, which means there wasn't any Read-Modify-Write going on in either case. As I am getting nearly the theoretical MAX, one might think there isn't a problem. However, when I increase TC to 5, my TP is only 750 MB/s. When I increase TC to 32, my TP is about 250 MB/s. While you may conclude that 32 threads is overwhelming the system, 5 threads shouldn't be overwhelming a system that has 12 core. When you consider that I get MAX TP with 4 threads, the fact that I get a 25% reduction by adding one thread is a problem. Bob Kierski Senior Storage Performance Engineer Cray Inc. 380 Jackson Street Suite 210 St. Paul, MN 55101 Tele: 651-967-9590 Fax: 651-605-9001 Cell: 651-890-7461 -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
