Hi, On Fri, 2011-03-18 at 01:42 -0400, Valeriu Mutu wrote: > Hi, > > Has anyone done any GFS2 metadata performance benchmarks? If so, what have you found? Also, what performance tuning would be recommended to increase the metadata performance of a GFS2 filesystem? > > I've recently ran 'fdtree' [1] against a GFS2 filesystem as well as an ext3 filesystem. Here's what I've found: > >From time to time, I run various test, but fdtree has not been among them. > (fdtree on a GFS2 filesystem) > # ./fdtree.bash -l 10 -d 3 -f 4 -s 1 -o /gfs2bench/fdtree > fdtree-1.0.2: starting at /gfs2bench/fdtree//LEVEL0.vm1.23787/ > creating/deleting 10 directory levels with 3 directories at each level > for a total of 88573 directories > with 4 files of size 4KiB per directory > for a total of 354292 files and 1417168KiB > Sun Mar 13 00:45:31 EST 2011 > Sun Mar 13 00:58:46 EST 2011 > DIRECTORY CREATE TIME IN, OUT, TOTAL = 0, 795, 795 > Directory creates per second = 111 > Sun Mar 13 00:58:46 EST 2011 > Sun Mar 13 03:00:44 EDT 2011 > FILE CREATE TIME IN, OUT, TOTAL = 795, 4513, 3718 > File creates per second = 95 > KiB per second = 381 > Sun Mar 13 03:00:44 EDT 2011 > Sun Mar 13 04:49:08 EDT 2011 > FILE REMOVE TIME IN, OUT, TOTAL = 4513, 11017, 6504 > File removals per second = 54 > Sun Mar 13 04:49:08 EDT 2011 > Sun Mar 13 05:02:58 EDT 2011 > DIRECTORY REMOVE TIME IN, OUT, TOTAL = 11017, 11847, 830 > Directory removals per second = 106 > > (fdtree on an ext3 filesystem) > # ./fdtree.bash -l 10 -d 3 -f 4 -s 1 -o /ext3bench/fdtree > fdtree-1.0.2: starting at /ext3bench/fdtree//LEVEL0.vm1.25896/ > creating/deleting 10 directory levels with 3 directories at each level > for a total of 88573 directories > with 4 files of size 4KiB per directory > for a total of 354292 files and 1417168KiB > Sun Mar 13 18:41:11 EDT 2011 > Sun Mar 13 18:45:48 EDT 2011 > DIRECTORY CREATE TIME IN, OUT, TOTAL = 0, 277, 277 > Directory creates per second = 319 > Sun Mar 13 18:45:49 EDT 2011 > Sun Mar 13 19:04:33 EDT 2011 > FILE CREATE TIME IN, OUT, TOTAL = 278, 1402, 1124 > File creates per second = 315 > KiB per second = 1260 > Sun Mar 13 19:04:33 EDT 2011 > Sun Mar 13 19:09:15 EDT 2011 > FILE REMOVE TIME IN, OUT, TOTAL = 1402, 1684, 282 > File removals per second = 1256 > Sun Mar 13 19:09:15 EDT 2011 > Sun Mar 13 19:10:42 EDT 2011 > DIRECTORY REMOVE TIME IN, OUT, TOTAL = 1684, 1771, 87 > Directory removals per second = 1018 > > In other words, ext3 is about 3 times faster at creating files/dirs, about 20 times faster at removing existing files, and about 10 times faster at removing existing directories. I've added the following lines to /etc/cluster/cluster.conf to remove the plock rate limit: > <dlm plock_ownership="1" plock_rate_limit="0"/> > <gfs_controld plock_rate_limit="0"/> > but this didn't help increase the GFS2 metadata performance. > These settings only affect the performance of the fcntl POSIX locks which do not have any "on disk" representation and are processed in userspace via gfs_controld/dlm_controld depending on which version you are using. So that is an expected result. > Note that I've used the same setup for the GFS2 and ext3 tests: same machine, same networking config, same storage array (which is not used by anything else). > I also confirmed using "pingpong" [2] that I get a rate of about 4K locks/sec on this particular node against GFS2. > The pingpong test does not test metadata performance. > Does anyone have any hints/ideas as what might help increase the metadata performance of a GFS2 filesystem? > > [1] https://computing.llnl.gov/?set=code&page=sio_downloads > [2] http://wiki.samba.org/index.php/Ping_pong > > Best, There are a number of variables which you don't mention, but which are important for the test results. Firstly, what kind of storage are you using? Secondly, was this lock_dlm or lock_nolock? Also was there any memory pressure while the tests were running? Was noatime set on the filesystem (or indeed, other mount options)? Steve. -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
