Hi there,
with FhGFS we may use extended attributes to store file meta data and
with ext3/ext4 that also works very well and the rate to create files
and to write those EAs (create() + fsetxattr() is about 2.5 to 3 times
faster than with a create() + write(). Size of those EA data is about
256 bytes depending on the number of storage stripes.
However, with XFS using extended attributes is *extremely* slow. Here
are some numbers with ext4 and xfs using a patched [1] bonnie++
schubert@fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestXFS/ -s0 -n 1:256:256:1 -r 0 -X
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fsdevel3 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1:256:256 330 6 +++++ +++ 47 1 469 9 +++++ +++ 31 1
Latency 962ms 36us 140ms 866ms 28us 311ms
1.96,1.96,fsdevel3,1,1314378941,,,,,,,,,,,,,,1,256,256,,,330,6,+++++,+++,47,1,469,9,+++++,+++,31,1,,,,,,,962ms,36us,140ms,866ms,28us,311ms
schubert@fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestEXT4/ -s0 -n 100:256:256:10 -r 0 -X
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fsdevel3 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100:256:256/10 21348 52 +++++ +++ 32878 61 25286 60 +++++ +++ 25873 53
Latency 746us 1926us 2553us 653us 118us 32250us
1.96,1.96,fsdevel3,1,1314379136,,,,,,,,,,,,,,100,256,256,,10,21348,52,+++++,+++,32878,61,25286,60,+++++,+++,25873,53,,,,,,,746us,1926us,2553us,653us,118us,32250us
NOTE: For ext4 I had to increase the number of files by factor 100 to
get any sane number (it would only print '+++++' otherwise). Running the
benchmark with the same numbers on xfs deleted in so slow delete
numbers, that it probably still would be busy to delete files by tomorrow.
The xfs file system was formated with these parameters:
mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/mapper/vg0fsdev3-XFStest
ext4 was formated to have an inode size of 512B and to have a joural
size of 400MB.
Both file systems are mounted with "nobarrier" (real FhGFS production
systems usually have raid controllers with battery backups).
Any ideas why xfs is so slow with extended attributes? Without extended
attributes, so skipping the "-X" option in our patched bonnie also
results in slower Create numbers (about 4500 on xfs vs. 7500 on ext4)
compared to ext4, but that is 'only' factor 1.67 and not 50 or more as
with EAs.
Thanks,
Bernd
[1] Updated bonnie to support extended attributes:
https://bitbucket.org/aakef/bonnie/
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
