On Wed, May 19, 2021 at 11:00:03AM +0300, Avi Kivity wrote: > > On 18/05/2021 02.22, Dave Chinner wrote: > > > > > What I'd like to do is remove the fanout directories, so that for each logical > > > "volume"[*] I have a single directory with all the files in it. But that > > > means sticking massive amounts of entries into a single directory and hoping > > > it (a) isn't too slow and (b) doesn't hit the capacity limit. > > Note that if you use a single directory, you are effectively single > > threading modifications to your file index. You still need to use > > fanout directories if you want concurrency during modification for > > the cachefiles index, but that's a different design criteria > > compared to directory capacity and modification/lookup scalability. > > Something that hit us with single-large-directory and XFS is that > XFS will allocate all files in a directory using the same > allocation group. If your entire filesystem is just for that one > directory, then that allocation group will be contended. There is more than one concurrency problem that can arise from using single large directories. Allocation policy is just another aspect of the concurrency picture. Indeed, you can avoid this specific problem simply by using the inode32 allocator - this policy round-robins files across allocation groups instead of trying to keep files physically local to their parent directory. Hence if you just want one big directory with lots of files that index lots of data, using the inode32 allocator will allow the files in the filesytsem to allocate/free space at maximum concurrency at all times... > We saw spurious ENOSPC when that happened, though that > may have related to bad O_DIRECT management by us. You should not see spurious ENOSPC at all. The only time I've recall this sort of thing occurring is when large extent size hints are abused by applying them to every single file and allocation regardless of whether they are needed, whilst simultaneously mixing long term and short term data in the same physical locality. Over time the repeated removal and reallocation of short term data amongst long term data fragments the crap out of free space until there are no large contiguous free spaces left to allocate contiguous extents from. > We ended up creating files in a temporary directory and moving them to the > main directory, since for us the directory layout was mandated by > compatibility concerns. inode32 would have done effectively the same thing but without needing to change the application.... > We are now happy with XFS large-directory management, but are nowhere close > to a million files. I think you are conflating directory scalability with problems arising from file allocation policies not being ideal for your data set organisation, layout and longevity characteristics. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
