On Fri Jan 11, 2013, Thomas Fjellstrom wrote: > On Thu Jan 10, 2013, Stan Hoeppner wrote: > > On 1/10/2013 3:36 PM, Chris Murphy wrote: > > > On Jan 10, 2013, at 3:49 AM, Thomas Fjellstrom <thomas@xxxxxxxxxxxxx> wrote: > > >> A lot of it will be streaming. Some may end up being random > > >> read/writes. The test is just to gauge over all performance of the > > >> setup. 600MBs read is far more than I need, but having writes at 1/3 > > >> that seems odd to me. > > > > > > Tell us how many disks there are, and what the chunk size is. It could > > > be too small if you have too few disks which results in a small full > > > stripe size for a video context. If you're using the default, it could > > > be too big and you're getting a lot of RWM. Stan, and others, can > > > better answer this. > > > > Thomas is using a benchmark, and a single one at that, to judge the > > performance. He's not using his actual workloads. Tuning/tweaking to > > increase the numbers in a benchmark could be detrimental to actual > > performance instead of providing a boost. One must be careful. > > > > Regarding RAID6, it will always have horrible performance compared to > > non-parity RAID levels and even RAID5, for anything but full stripe > > aligned writes, which means writing new large files or doing large > > appends to existing files. > > Considering its a rather simple use case, mostly streaming video, and misc > file sharing for my home network, an iozone test should be rather telling. > Especially the full test, from 4k up to 16mb > > random random > bkwd record stride KB reclen write rewrite read reread > read write read rewrite read fwrite frewrite fread freread > 33554432 4 243295 221756 628767 624081 1028 4627 16822 > 7468777 17740 233295 231092 582036 579131 33554432 8 > 241134 225728 628264 627015 2027 8879 25977 10030302 19578 > 228923 233928 591478 584892 33554432 16 233758 228122 > 633406 618248 3952 13635 35676 10166457 19968 227599 > 229698 579267 576850 33554432 32 232390 219484 625968 625627 > 7604 18800 44252 10728450 24976 216880 222545 556513 > 555371 33554432 64 222936 206166 631659 627823 14112 22837 > 52259 11243595 30251 196243 192755 498602 494354 33554432 > 128 214740 182619 628604 626407 25088 26719 64912 11232068 > 39867 198638 185078 463505 467853 33554432 256 202543 185964 > 626614 624367 44363 34763 73939 10148251 62349 176724 > 191899 593517 595646 33554432 512 208081 188584 632188 629547 > 72617 39145 84876 9660408 89877 182736 172912 610681 > 608870 33554432 1024 196429 166125 630785 632413 116793 51904 > 133342 8687679 121956 168756 175225 620587 616722 33554432 > 2048 185399 167484 622180 627606 188571 70789 218009 5357136 > 370189 171019 166128 637830 637120 33554432 4096 198340 188695 > 632693 628225 289971 95211 278098 4836433 611529 161664 > 170469 665617 655268 33554432 8192 177919 167524 632030 629077 > 371602 115228 384030 4934570 618061 161562 176033 708542 > 709788 33554432 16384 196639 183744 631478 627518 485622 133467 > 462861 4890426 644615 175411 179795 725966 734364 > > > However, everything is relative. This RAID6 may have plenty of random > > and streaming write/read throughput for Thomas. But a single benchmark > > isn't going to inform him accurately. > > 200MB/s may be enough, but the difference between the read and write > throughput is a bit unexpected. It's not a weak machine (core i3-2120, dual > core 3.2Ghz with HT, 16GB ECC 1333Mhz ram), and this is basically all its > going to be doing. > > > > You said these are unpartitioned disks, I think. In which case > > > alignment of 4096 byte sectors isn't a factor if these are AF disks. > > > > > > Unlikely to make up the difference is the scheduler. Parallel fs's like > > > XFS don't perform nearly as well with CFQ, so you should have a kernel > > > parameter elevator=noop. > > > > If the HBAs have [BB|FB]WC then one should probably use noop as the > > cache schedules the actual IO to the drives. If the HBAs lack cache, > > then deadline often provides better performance. Testing of each is > > required on a system and workload basis. With two identical systems > > (hardware/RAID/OS) one may perform better with noop, the other with > > deadline. The determining factor is the applications' IO patterns. > > Mostly streaming reads, some long rsync's to copy stuff back and forth, > file share duties (downloads etc). > > > > Another thing to look at is md/stripe_cache_size which probably needs > > > to be higher for your application. > > > > > > Another thing to look at is if you're using XFS, what your mount > > > options are. Invariably with an array of this size you need to be > > > mounting with the inode64 option. > > > > The desired allocator behavior is independent of array size but, once > > again, dependent on the workloads. inode64 is only needed for large > > filesystems with lots of files, where 1TB may not be enough for the > > directory inodes. Or, for mixed metadata/data heavy workloads. > > > > For many workloads including databases, video ingestion, etc, the > > inode32 allocator is preferred, regardless of array size. This is the > > linux-raid list so I'll not go into detail of the XFS allocators. > > If you have the time and the desire, I'd like to hear about it off list. > > > >> The reason I've selected RAID6 to begin with is I've read (on this > > >> mailing list, and on some hardware tech sites) that even with SAS > > >> drives, the rebuild/resync time on a large array using large disks > > >> (2TB+) is long enough that it gives more than enough time for another > > >> disk to hit a random read error, > > > > > > This is true for high density consumer SATA drives. It's not nearly as > > > applicable for low to moderate density nearline SATA which has an order > > > of magnitude lower UER, or for enterprise SAS (and some enterprise > > > SATA) which has yet another order of magnitude lower UER. So it > > > depends on the disks, and the RAID size, and the backup/restore > > > strategy. > > > > Yes, enterprise drives have a much larger spare sector pool. > > > > WRT rebuild time, this is one more reason to use RAID10 or a concat of > > RAID1s. The rebuild time is low, constant, predictable. For 2TB drives > > about 5-6 hours at 100% rebuild rate. And rebuild time, for any array > > type, with gargantuan drives, is yet one more reason not to use the > > largest drives you can get your hands on. Using 1TB drives will cut > > that to 2.5-3 hours, and using 500GB drives will cut it down to 1.25-1.5 > > hours, as all these drives tend to have similar streaming write rates. > > > > To wit, as a general rule I always build my arrays with the smallest > > drives I can get away with for the workload at hand. Yes, for a given > > TB total it increases acquisition cost of drives, HBAs, enclosures, and > > cables, and power consumption, but it also increases spindle count--thus > > performance-- while decreasing rebuild times substantially/dramatically. > > I'd go raid10 or something if I had the space, but this little 10TB nas > (which is the goal, a small, quiet, not too slow, 10TB nas with some kind > of redundancy) only fits 7 3.5" HDDs. > > Maybe sometime in the future I'll get a big 3 or 4 u case with a crap load > of 3.5" HDD bays, but for now, this is what I have (as well as my old > array, 7x1TB RAID5+XFS in 4in3 hot swap bays with room for 8 drives, but > haven't bothered to expand the old array, and I have the new one almost > ready to go). > > I don't know if it impacts anything at all, but when burning in these > drives after I bought them, I ran the same full iozone test a couple > times, and each drive shows 150MB/s read, and similar write times > (100-120+?). It impressed me somewhat, to see a mechanical hard drive go > that fast. I remember back a few years ago thinking 80MBs was fast for a > HDD. I should note, it might do some p2p duties in the future. Not sure about that. -- Thomas Fjellstrom thomas@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
