Re: How capacious and well-indexed are ext4, xfs and btrfs directories?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 18/05/2021 02.22, Dave Chinner wrote:

What I'd like to do is remove the fanout directories, so that for each logical
"volume"[*] I have a single directory with all the files in it.  But that
means sticking massive amounts of entries into a single directory and hoping
it (a) isn't too slow and (b) doesn't hit the capacity limit.
Note that if you use a single directory, you are effectively single
threading modifications to your file index. You still need to use
fanout directories if you want concurrency during modification for
the cachefiles index, but that's a different design criteria
compared to directory capacity and modification/lookup scalability.


Something that hit us with single-large-directory and XFS is that XFS will allocate all files in a directory using the same allocation group. If your entire filesystem is just for that one directory, then that allocation group will be contended. We saw spurious ENOSPC when that happened, though that may have related to bad O_DIRECT management by us.


We ended up creating files in a temporary directory and moving them to the main directory, since for us the directory layout was mandated by compatibility concerns.


We are now happy with XFS large-directory management, but are nowhere close to a million files.





[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]

  Powered by Linux