I'm curious what filesystem you would recommend for his use-case. -RZp.s. Nicolas' users are only slightly "angry" compared to mine... My users are well and truly MAD, however. Think 16-TB ext4 filesystems with 400,000,000 files smaller than 64K.
On 11/07/2011, Alan Brown <ajb2@xxxxxxxxxxxxxx> wrote:
Nicolas Ross wrote:On some services, there are document directories that are huge, not that much in size (about 35 gigs), but in number of files, around one million. One service even has 3 data directories with that many files each.You are utterly mad. Apart from the human readability aspects if someone attempts a directory listing, you're putting a substantial load on your system each time you attempt to go into those directories, even with dentry/inode caching tweaked out to maximums. Directory inode hashing helps, but not for filesystem abuse on this scale. Be glad you're using ext3/4 and not GFS, the problems are several orders of magnitude worse there (it can take 10 minutes to list a directory with 10,000 files in it, let alone 1,000,000) > It works pretty well for now, but when it comes to data update (viarsync) and backup (also via rsync), the node doing the rsync crawls to a stop, all 16 logical cores are used at 100% system, and it sometimes freezes the file system for other services on other nodes.That's not particularly surprising - and a fairly solid hint you should be revisiting the way you lay out your files. If you go for a hierarchical layout you'll see several orders of magnitude speedup in access time without any real effort at all. If you absolutely must put that many files in a directory, then use a filesystem able to cope with such activities. Ext3/4 aren't it.
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
