Sean McCauliff <smccauliff@xxxxxxxxxxxxxxxxx> wrote: >I plan on having about 100M files totaling about 8.5TiBytes. To see >how ext3 would perform with large numbers of files I've written a test >program which creates a configurable number of files into a configurable >number of directories, reads from those files, lists them and then >deletes them. Even up to 1M files ext3 seems to perform well and scale >linearly; the time to execute the program on 1M files is about double >the time it takes it to execute on .5M files. But past 1M files it >seems to have n^2 scalability. Test details appear below. > >Looking at the various options for ext3 nothing jumps out as the obvious >one to use to improve performance. Try increasing your journal size (mke2fs -J size=400), and having a lot of RAM. When you say "having about 100M files", does that mean "need to be constantly accessing 100M files" or just "need to store a total of 100M files in this filesystem"? The former means you need to keep the whole working set in RAM for maximum performance, about 100M * (128 + 32) = 19GB of RAM. The latter is no problem, we have ext3 filesystems with > 250M files in them. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. _______________________________________________ Ext3-users mailing list Ext3-users@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/ext3-users
