On Sun Jan 13, 2013, Tommy Apel Hansen wrote: > Could you do me a favor and run the iozone test with the -I switch on so > that we can seen the actual speed of the array and not you RAM Sure. Though I thought running the test with a file size twice the size of ram would help with that issue. > /Tommy > > On Fri, 2013-01-11 at 05:35 -0700, Thomas Fjellstrom wrote: > > On Thu Jan 10, 2013, Stan Hoeppner wrote: > > > On 1/10/2013 3:36 PM, Chris Murphy wrote: > > > > On Jan 10, 2013, at 3:49 AM, Thomas Fjellstrom <thomas@xxxxxxxxxxxxx> wrote: > > > >> A lot of it will be streaming. Some may end up being random > > > >> read/writes. The test is just to gauge over all performance of the > > > >> setup. 600MBs read is far more than I need, but having writes at > > > >> 1/3 that seems odd to me. > > > > > > > > Tell us how many disks there are, and what the chunk size is. It > > > > could be too small if you have too few disks which results in a > > > > small full stripe size for a video context. If you're using the > > > > default, it could be too big and you're getting a lot of RWM. Stan, > > > > and others, can better answer this. > > > > > > Thomas is using a benchmark, and a single one at that, to judge the > > > performance. He's not using his actual workloads. Tuning/tweaking to > > > increase the numbers in a benchmark could be detrimental to actual > > > performance instead of providing a boost. One must be careful. > > > > > > Regarding RAID6, it will always have horrible performance compared to > > > non-parity RAID levels and even RAID5, for anything but full stripe > > > aligned writes, which means writing new large files or doing large > > > appends to existing files. > > > > Considering its a rather simple use case, mostly streaming video, and > > misc file sharing for my home network, an iozone test should be rather > > telling. Especially the full test, from 4k up to 16mb > > > > random random > > bkwd record > > stride > > > > KB reclen write rewrite read reread read > > write read rewrite read fwrite frewrite fread > > freread > > > > 33554432 4 243295 221756 628767 624081 1028 > > 4627 16822 7468777 17740 233295 231092 582036 > > 579131 33554432 8 241134 225728 628264 627015 > > 2027 8879 25977 10030302 19578 228923 233928 591478 > > 584892 33554432 16 233758 228122 633406 618248 > > 3952 13635 35676 10166457 19968 227599 229698 579267 > > 576850 33554432 32 232390 219484 625968 625627 > > 7604 18800 44252 10728450 24976 216880 222545 556513 > > 555371 33554432 64 222936 206166 631659 627823 > > 14112 22837 52259 11243595 30251 196243 192755 > > 498602 494354 33554432 128 214740 182619 628604 > > 626407 25088 26719 64912 11232068 39867 198638 > > 185078 463505 467853 33554432 256 202543 185964 > > 626614 624367 44363 34763 73939 10148251 62349 > > 176724 191899 593517 595646 33554432 512 208081 > > 188584 632188 629547 72617 39145 84876 9660408 > > 89877 182736 172912 610681 608870 33554432 1024 > > 196429 166125 630785 632413 116793 51904 133342 > > 8687679 121956 168756 175225 620587 616722 33554432 > > 2048 185399 167484 622180 627606 188571 70789 218009 > > 5357136 370189 171019 166128 637830 637120 33554432 > > 4096 198340 188695 632693 628225 289971 95211 278098 > > 4836433 611529 161664 170469 665617 655268 33554432 > > 8192 177919 167524 632030 629077 371602 115228 384030 > > 4934570 618061 161562 176033 708542 709788 33554432 > > 16384 196639 183744 631478 627518 485622 133467 462861 > > 4890426 644615 175411 179795 725966 734364 > > > > > > However, everything is relative. This RAID6 may have plenty of random > > > and streaming write/read throughput for Thomas. But a single benchmark > > > isn't going to inform him accurately. > > > > 200MB/s may be enough, but the difference between the read and write > > throughput is a bit unexpected. It's not a weak machine (core i3-2120, > > dual core 3.2Ghz with HT, 16GB ECC 1333Mhz ram), and this is basically > > all its going to be doing. > > > > > > You said these are unpartitioned disks, I think. In which case > > > > alignment of 4096 byte sectors isn't a factor if these are AF disks. > > > > > > > > Unlikely to make up the difference is the scheduler. Parallel fs's > > > > like XFS don't perform nearly as well with CFQ, so you should have a > > > > kernel parameter elevator=noop. > > > > > > If the HBAs have [BB|FB]WC then one should probably use noop as the > > > cache schedules the actual IO to the drives. If the HBAs lack cache, > > > then deadline often provides better performance. Testing of each is > > > required on a system and workload basis. With two identical systems > > > (hardware/RAID/OS) one may perform better with noop, the other with > > > deadline. The determining factor is the applications' IO patterns. > > > > Mostly streaming reads, some long rsync's to copy stuff back and forth, > > file share duties (downloads etc). > > > > > > Another thing to look at is md/stripe_cache_size which probably needs > > > > to be higher for your application. > > > > > > > > Another thing to look at is if you're using XFS, what your mount > > > > options are. Invariably with an array of this size you need to be > > > > mounting with the inode64 option. > > > > > > The desired allocator behavior is independent of array size but, once > > > again, dependent on the workloads. inode64 is only needed for large > > > filesystems with lots of files, where 1TB may not be enough for the > > > directory inodes. Or, for mixed metadata/data heavy workloads. > > > > > > For many workloads including databases, video ingestion, etc, the > > > inode32 allocator is preferred, regardless of array size. This is the > > > linux-raid list so I'll not go into detail of the XFS allocators. > > > > If you have the time and the desire, I'd like to hear about it off list. > > > > > >> The reason I've selected RAID6 to begin with is I've read (on this > > > >> mailing list, and on some hardware tech sites) that even with SAS > > > >> drives, the rebuild/resync time on a large array using large disks > > > >> (2TB+) is long enough that it gives more than enough time for > > > >> another disk to hit a random read error, > > > > > > > > This is true for high density consumer SATA drives. It's not nearly > > > > as applicable for low to moderate density nearline SATA which has an > > > > order of magnitude lower UER, or for enterprise SAS (and some > > > > enterprise SATA) which has yet another order of magnitude lower UER. > > > > So it depends on the disks, and the RAID size, and the > > > > backup/restore strategy. > > > > > > Yes, enterprise drives have a much larger spare sector pool. > > > > > > WRT rebuild time, this is one more reason to use RAID10 or a concat of > > > RAID1s. The rebuild time is low, constant, predictable. For 2TB > > > drives about 5-6 hours at 100% rebuild rate. And rebuild time, for > > > any array type, with gargantuan drives, is yet one more reason not to > > > use the largest drives you can get your hands on. Using 1TB drives > > > will cut that to 2.5-3 hours, and using 500GB drives will cut it down > > > to 1.25-1.5 hours, as all these drives tend to have similar streaming > > > write rates. > > > > > > To wit, as a general rule I always build my arrays with the smallest > > > drives I can get away with for the workload at hand. Yes, for a given > > > TB total it increases acquisition cost of drives, HBAs, enclosures, and > > > cables, and power consumption, but it also increases spindle > > > count--thus performance-- while decreasing rebuild times > > > substantially/dramatically. > > > > I'd go raid10 or something if I had the space, but this little 10TB nas > > (which is the goal, a small, quiet, not too slow, 10TB nas with some > > kind of redundancy) only fits 7 3.5" HDDs. > > > > Maybe sometime in the future I'll get a big 3 or 4 u case with a crap > > load of 3.5" HDD bays, but for now, this is what I have (as well as my > > old array, 7x1TB RAID5+XFS in 4in3 hot swap bays with room for 8 drives, > > but haven't bothered to expand the old array, and I have the new one > > almost ready to go). > > > > I don't know if it impacts anything at all, but when burning in these > > drives after I bought them, I ran the same full iozone test a couple > > times, and each drive shows 150MB/s read, and similar write times > > (100-120+?). It impressed me somewhat, to see a mechanical hard drive go > > that fast. I remember back a few years ago thinking 80MBs was fast for a > > HDD. -- Thomas Fjellstrom thomas@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
