On 24/10/13 08:23, Stan Hoeppner wrote: > On 10/23/2013 2:03 AM, David Brown wrote: > >> On the other hand, he is also serving 100+ freenx desktop users. As far >> as I understand it (and I'm very happy for corrections if I'm wrong), >> that will mean a /home directory with 100+ sub-directories for the >> different users - and that /is/ one of the ideal cases for concat+XFS >> parallelism. > > No, it is /not/. Homedir storage is not an ideal use case. It's not > even in the ballpark. There's simply not enough parallelism nor IOPS > involved, and file sizes can vary substantially, so the workload is not > deterministic, i.e. it is "general". Recall I said in my last reply > that this "is a very workload specific storage architecture"? > > Workloads that benefit from XFS over concatenated disks are those that: > > 1. Expose inherent limitations and/or inefficiencies of striping, > at the filesystem, elevator, and/or hardware level > > 2. Exhibit a high degree of directory level parallelism > > 3. Exhibit high IOPS or data rates > > 4. Most importantly, exhibit relatively deterministic IO patterns > > Typical homedir storage meets none of these criteria. Homedir files on > a GUI desktop terminal server are not 'typical', but the TS workload > doesn't meet these criteria either. > I am trying to learn from your experience and knowledge here, so thank you for your time so far. Hopefully it is also of use and interest to others - that's one of the beauties of public mailing lists. Am I correct in thinking that a common "ideal use case" is a mail server with lots of accounts, especially with maildir structures, so that accesses are spread across lots of directories with typically many parallel accesses to many small files? First, to make sure I am not making any technical errors here, I believe that when you make your XFS over a linear concat, the allocation groups are spread evenly across the parts of the concat so that logically (by number) adjacent AG's will be on different underlying disks. When you make a new directory on the filesystem, it gets put in a different AG (wrapping around, of course, and overflowing when necessary). Thus if you make three directories, and put a file in each directory, then each file will be on a different disk. (I believe older XFS only allocated different top-level directories to different AG's, but current XFS does so for all directories). I have been thinking about what the XFS over concat gives you compared to XFS over raid0 on the same disks (or raid1 pairs - the details don't matter much). First, consider small files. Access to small files (smaller than the granularity of the raid0 chunks) will usually only involve one disk of the raid0 stripe, and will /definitely/ only involve one disk of the concat. You should be able to access multiple small files in parallel, if you are lucky in the mix (with raid0, this "luck" will be mostly random, while with concat it will depend on the mix of files within directories. In particular, multiple files within the same directory will not be paralleled). With a concat, all relevant accesses such as directory reads and inode table access will be within the same disk as the file, while with raid0 it could easily be a different disk - but such accesses are often cached in ram. With raid0 you have the chance of the small file spanning two disks, leading to longer latency for that file and for other parallel accesses. All in all, small file access should not be /too/ different - but my guess is concat has the edge for lowest overall latency with multiple parallel accesses, as I think concat will avoid jumps between disks better. For large files, there is a bigger difference. Raid0 gives striping for higher throughput - but these accesses block the parallel accesses to other files. concat has slower throughput as there is no striping, but the other disks are free for parallel accesses (big or small). To my mind, this boils down to a question of balancing - concat gives lower average latencies with highly parallel accesses, but sacrifices maximum throughput of large files. If you don't have lots of parallel accesses, then concat gains little or nothing compared to raid0. If I try to match up this with the points you made, point 1 about striping is clear - this is a major difference between concat and raid0. Point 2 and 3 about parallelism and high IOPs (and therefore low latency) is also clear - if you don't need such access, concat will give you nothing. Only the OP can decide if his usage will meet these points. But I am struggling with point 4 - "most importantly, exhibit relatively deterministic IO patterns". All you need is to have your file accesses spread amongst a range of directories. If the number of (roughly) parallel accesses is big enough, you'll get a fairly even spread across the disks - and if it is not big enough for that, you haven't matched point 2. This is not really much different from raid0 - small accesses will be scattered across the different disks. The big difference comes when there is a large file access - with raid0, you will block /all/ other accesses for a time, while with concat (over three disks) you will block one third of the accesses for three times as long. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
