Hi,
Has anyone done any GFS2 metadata performance benchmarks? If so, what have you found? Also, what performance tuning would be recommended to increase the metadata performance of a GFS2 filesystem?
I've recently ran 'fdtree' [1] against a GFS2 filesystem as well as an ext3 filesystem. Here's what I've found:
(fdtree on a GFS2 filesystem)
# ./fdtree.bash -l 10 -d 3 -f 4 -s 1 -o /gfs2bench/fdtree
fdtree-1.0.2: starting at /gfs2bench/fdtree//LEVEL0.vm1.23787/
creating/deleting 10 directory levels with 3 directories at each level
for a total of 88573 directories
with 4 files of size 4KiB per directory
for a total of 354292 files and 1417168KiB
Sun Mar 13 00:45:31 EST 2011
Sun Mar 13 00:58:46 EST 2011
DIRECTORY CREATE TIME IN, OUT, TOTAL = 0, 795, 795
Directory creates per second = 111
Sun Mar 13 00:58:46 EST 2011
Sun Mar 13 03:00:44 EDT 2011
FILE CREATE TIME IN, OUT, TOTAL = 795, 4513, 3718
File creates per second = 95
KiB per second = 381
Sun Mar 13 03:00:44 EDT 2011
Sun Mar 13 04:49:08 EDT 2011
FILE REMOVE TIME IN, OUT, TOTAL = 4513, 11017, 6504
File removals per second = 54
Sun Mar 13 04:49:08 EDT 2011
Sun Mar 13 05:02:58 EDT 2011
DIRECTORY REMOVE TIME IN, OUT, TOTAL = 11017, 11847, 830
Directory removals per second = 106
(fdtree on an ext3 filesystem)
# ./fdtree.bash -l 10 -d 3 -f 4 -s 1 -o /ext3bench/fdtree
fdtree-1.0.2: starting at /ext3bench/fdtree//LEVEL0.vm1.25896/
creating/deleting 10 directory levels with 3 directories at each level
for a total of 88573 directories
with 4 files of size 4KiB per directory
for a total of 354292 files and 1417168KiB
Sun Mar 13 18:41:11 EDT 2011
Sun Mar 13 18:45:48 EDT 2011
DIRECTORY CREATE TIME IN, OUT, TOTAL = 0, 277, 277
Directory creates per second = 319
Sun Mar 13 18:45:49 EDT 2011
Sun Mar 13 19:04:33 EDT 2011
FILE CREATE TIME IN, OUT, TOTAL = 278, 1402, 1124
File creates per second = 315
KiB per second = 1260
Sun Mar 13 19:04:33 EDT 2011
Sun Mar 13 19:09:15 EDT 2011
FILE REMOVE TIME IN, OUT, TOTAL = 1402, 1684, 282
File removals per second = 1256
Sun Mar 13 19:09:15 EDT 2011
Sun Mar 13 19:10:42 EDT 2011
DIRECTORY REMOVE TIME IN, OUT, TOTAL = 1684, 1771, 87
Directory removals per second = 1018
In other words, ext3 is about 3 times faster at creating files/dirs, about 20 times faster at removing existing files, and about 10 times faster at removing existing directories. I've added the following lines to /etc/cluster/cluster.conf to remove the plock rate limit:
<dlm plock_ownership="1" plock_rate_limit="0"/>
<gfs_controld plock_rate_limit="0"/>
but this didn't help increase the GFS2 metadata performance.
Note that I've used the same setup for the GFS2 and ext3 tests: same machine, same networking config, same storage array (which is not used by anything else).
I also confirmed using "pingpong" [2] that I get a rate of about 4K locks/sec on this particular node against GFS2.
Does anyone have any hints/ideas as what might help increase the metadata performance of a GFS2 filesystem?
[1] https://computing.llnl.gov/?set=code&page=sio_downloads
[2] http://wiki.samba.org/index.php/Ping_pong
Best,
--
Valeriu Mutu
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
