On Sat, Mar 12, 2011 at 10:14 PM, Nico Kadel-Garcia <nkadel@xxxxxxxxx> wrote:
> On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux <aspineux@xxxxxxxxx> wrote:
>> Thanks to everybody for answering.
>>
>> I thing >250E6 is a lot and keep decent read and write access speed is unreal
>> using mutli-purpose filesystems like ext? and other ?FS.
>> I would need a dedicated filesystem for that.
>>
>> This problem was only a possible solution to another problem.
>> I will solve the original problem using another way.
>>
>
> Why, exactly, are you doing this? The normal approach for such dense
> repositories is to create a hierarchy of subdirectories.
>
> File aaa12345 goes in
>
> $DIR/a/a/a/12345
>
> File abc6789 goes in
>
> $DIR/a/b/c/6789
Try to create this king of tree yourself and when done, remove it.
I took hours on my box and it is even faster to keep all files in the
same diretory.
I looks like working in multiple directories slow down the process also
I read other posts and articles and handle more than 100M files become
a problem !
256M is a problem and more than 1G files is a big problem.
I was splitting data into files to help me. I will keep them in big files.
Regards
Anyway it was too slow.
>
> And whatever is accessing or creating the files is taught the
> algorithm used. This requires some programming up front, but helps
> prevent precisely the outrageous directory size you describe.
> Handling, sorting, and reporting on that many files in one directory
> is an old and painful problem.
> _______________________________________________
> CentOS mailing list
> CentOS@xxxxxxxxxx
> http://lists.centos.org/mailman/listinfo/centos
>
--
Alain Spineux | aspineux gmail com
Monitor your iT & Backups | http://www.magikmon.com
Free Backup front-end | http://www.magikmon.com/mksbackup
Your email 100% available | http://www.emailgency.com
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
