On 10/24/2013 2:26 AM, David Brown wrote: > On 24/10/13 08:23, Stan Hoeppner wrote: >> On 10/23/2013 2:03 AM, David Brown wrote: >> >>> On the other hand, he is also serving 100+ freenx desktop users. As far >>> as I understand it (and I'm very happy for corrections if I'm wrong), >>> that will mean a /home directory with 100+ sub-directories for the >>> different users - and that /is/ one of the ideal cases for concat+XFS >>> parallelism. >> >> No, it is /not/. Homedir storage is not an ideal use case. It's not >> even in the ballpark. There's simply not enough parallelism nor IOPS >> involved, and file sizes can vary substantially, so the workload is not >> deterministic, i.e. it is "general". Recall I said in my last reply >> that this "is a very workload specific storage architecture"? >> >> Workloads that benefit from XFS over concatenated disks are those that: >> >> 1. Expose inherent limitations and/or inefficiencies of striping, >> at the filesystem, elevator, and/or hardware level >> >> 2. Exhibit a high degree of directory level parallelism >> >> 3. Exhibit high IOPS or data rates >> >> 4. Most importantly, exhibit relatively deterministic IO patterns >> >> Typical homedir storage meets none of these criteria. Homedir files on >> a GUI desktop terminal server are not 'typical', but the TS workload >> doesn't meet these criteria either. If you could sum up everything below into a couple of short, direct, coherent questions you have, I'd be glad to address them. > I am trying to learn from your experience and knowledge here, so thank > you for your time so far. Hopefully it is also of use and interest to > others - that's one of the beauties of public mailing lists. > > > Am I correct in thinking that a common "ideal use case" is a mail server > with lots of accounts, especially with maildir structures, so that > accesses are spread across lots of directories with typically many > parallel accesses to many small files? > > > First, to make sure I am not making any technical errors here, I believe > that when you make your XFS over a linear concat, the allocation groups > are spread evenly across the parts of the concat so that logically (by > number) adjacent AG's will be on different underlying disks. When you > make a new directory on the filesystem, it gets put in a different AG > (wrapping around, of course, and overflowing when necessary). Thus if > you make three directories, and put a file in each directory, then each > file will be on a different disk. (I believe older XFS only allocated > different top-level directories to different AG's, but current XFS does > so for all directories). > > > > I have been thinking about what the XFS over concat gives you compared > to XFS over raid0 on the same disks (or raid1 pairs - the details don't > matter much). > > First, consider small files. Access to small files (smaller than the > granularity of the raid0 chunks) will usually only involve one disk of > the raid0 stripe, and will /definitely/ only involve one disk of the > concat. You should be able to access multiple small files in parallel, > if you are lucky in the mix (with raid0, this "luck" will be mostly > random, while with concat it will depend on the mix of files within > directories. In particular, multiple files within the same directory > will not be paralleled). With a concat, all relevant accesses such as > directory reads and inode table access will be within the same disk as > the file, while with raid0 it could easily be a different disk - but > such accesses are often cached in ram. With raid0 you have the chance > of the small file spanning two disks, leading to longer latency for that > file and for other parallel accesses. > > All in all, small file access should not be /too/ different - but my > guess is concat has the edge for lowest overall latency with multiple > parallel accesses, as I think concat will avoid jumps between disks better. > > > For large files, there is a bigger difference. Raid0 gives striping for > higher throughput - but these accesses block the parallel accesses to > other files. concat has slower throughput as there is no striping, but > the other disks are free for parallel accesses (big or small). > > > To my mind, this boils down to a question of balancing - concat gives > lower average latencies with highly parallel accesses, but sacrifices > maximum throughput of large files. If you don't have lots of parallel > accesses, then concat gains little or nothing compared to raid0. > > > If I try to match up this with the points you made, point 1 about > striping is clear - this is a major difference between concat and raid0. > Point 2 and 3 about parallelism and high IOPs (and therefore low > latency) is also clear - if you don't need such access, concat will give > you nothing. > > Only the OP can decide if his usage will meet these points. > > But I am struggling with point 4 - "most importantly, exhibit relatively > deterministic IO patterns". All you need is to have your file accesses > spread amongst a range of directories. If the number of (roughly) > parallel accesses is big enough, you'll get a fairly even spread across > the disks - and if it is not big enough for that, you haven't matched > point 2. This is not really much different from raid0 - small accesses > will be scattered across the different disks. The big difference comes > when there is a large file access - with raid0, you will block /all/ > other accesses for a time, while with concat (over three disks) you will > block one third of the accesses for three times as long. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
