On Wed, 8 Jul 2009, oooooooooooo ooooooooooooo wrote:
>
> Hi,
>
> I have a program that writes lots of files to a directory tree (around 15 Million fo files), and a node can have up to 400000 files (and I don't have any way to split this ammount in smaller ones). As the number of files grows, my application gets slower and slower (the app is works something like a cache for another app and I can't redesign the way it distributes files into disk due to the other app requirements).
>
> The filesystem I use is ext3 with teh following options enabled:
>
> Filesystem features: has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
>
> Is there any way to improve performance in ext3? Would you suggest another FS for this situation (this is a prodution server, so I need a stable one) ?
>
> Thanks in advance (and please excuse my bad english).
There isn't a good file system for this type of thing. filesystems with
many very small files are always slow. Ext3, XFS, JFS are all terrible
for this type of thing.
Rethink how you're writing files or you'll be in a world of hurt.
--
James A. Peltier
Systems Analyst (FASNet), VIVARIUM Technical Director
HPC Coordinator
Simon Fraser University - Burnaby Campus
Phone : 778-782-6573
Fax : 778-782-3045
E-Mail : jpeltier@xxxxxx
Website : http://www.fas.sfu.ca | http://vivarium.cs.sfu.ca
http://blogs.sfu.ca/people/jpeltier
MSN : subatomic_spam@xxxxxxxxxxx
The point of the HPC scheduler is to
keep everyone equally unhappy.
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
