On 09/26/2013 09:12 AM, Ronnie Tartar wrote:
Stan,
Thanks for the reply.
My fragmentation is:
[root@AP-FS1 ~]# xfs_db -c frag -r /dev/xvdb1
actual 10470159, ideal 10409782, fragmentation factor 0.58%
This was never likely the cause ...
[...]
This is running virtualized, definitely not a rust bucket. It's x5570 cpus
... well, this is likely the cause (virtualized)
with MD3200 Array with light I/O.
Seems like i/o wait is not problem, system% is problem. Is this the OS
trying to find spot for these files?
From your previous description
takes. The folders are image folders that have anywhere between 5 to
10 million images in each folder.
The combination of very large folders, and virtualization is working
against you. Couple that with an old (ancient by Linux standards) xfs
in the virtual CentOS 5.9 system, and you aren't going to have much joy
with this without changing a few things.
First and foremost:
Can you change from one single large folder to a heirarchical set of
folders? The single large folder means any metadata operation (ls,
stat, open, close) has a huge set of lists to traverse. It will work,
albiet slowly. As a rule of thumb, we try to make sure our users don't
go much beyond 10k files/folder. If they need to, building a heirarchy
of folders slightly increases management complexity, but keeps the lists
that are needed to be traversed much smaller.
A strategy for doing this: If your files are named "aaaa0001"
"aaaa0002" ... "zzzz9999" or similar, then you can chop off the first
letter, and make a directory of it, and then put all files starting with
that letter in that directory. Then within each of those directories,
do the same thing with the second letter. This gets you 676 directories
and about 15k files per directory. Much faster directory operations.
Much smaller lists to traverse.
If you can't change the layout, this is a much harder problem to solve,
though you could do it by using one very large file, maintaining your
own start/end and file metadata in other files, and writing your files
at specific offsets (start and end above). This isn't a good solution
unless you know how to write file systems.
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
