Re: How to find a needle in a haystack?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



>> I have a backup server that contains 10 ext3 file systems each with 12
>> million files scattered randomly over 4000 directories.  The files
>> average
>
> OK 4000 directories for 12million files means you've got 3000 files per
> directory. If you are doing that make sure your fs has htree enabled. The
> initial find is still going to suck especially if on spinning media and
> doubly (actually far more than doubly) if the disk is also busy trying to
> do other work.

dir_index is enabled as is noatime.  I'm not sure but I believe the
noatime was supposed to speed things up a little also.

>> size is 1MB.  Every day I expect to get 20 or so requests for files from
>> this archive.  The files were not stored in any logical structure that I
>> can use to narrow down the search.  This will be different moving
>> forward
>> but it does not help me for the old data.  Additionally, every day data
>> is
>> added and old data is removed to make space.
>
> What are you searching by - name or content ?
>
> If you are searching by content then you really want to stuff the lot
> into a freetext engine like Omega. If you are searching by file name I'd
> take the time to turn the archive upside down and archive that way up
> assuming your tools can do it.
>
> That is turn all the files backwards so
>
> 	foo/bar/hello.c
>
> becomes
>
> 	hello.c/bar/foo
>
> or build a symlink farm of them that way up (so hello.c/bar/foo is a
> symlink to foo/bar/hello.c)
>
> Its then suddenely a lot lot less painful to find things ! A database
> would also no doubt do the job

I am searching by name only.  I am a little unclear about this.  The idea
here is to create a symlink for each file in the root of the filesystem? 
That won't hurt something?

---
Will Y.


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

-- 
users mailing list
users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [EPEL Devel]     [Fedora Magazine]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Desktop]     [Fedora Fonts]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Fedora Sparc]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux