In followup on this thread,
We discovered that the reason why we didn't notice this performance
problem in our initial proof of concept was because some directories
were recently added to our source repository which contain a large
number of files and subdirectories. I tested a number of directories
and discovered that other similar sized directories (1 to 2 G) could
remove in 1 to 3 seconds but the 1.6 G directory referenced below took
~ 9 minutes. This particular directory contained ~ 10,000
subdirectories containing a total of 65,000+ files.
Further tests with touching empty files revealed that a directory with
65,000 empty files removed relatively quickly while a directory with
65,000 subdirectories containing 1 empty file each took a very long
time to remove. Using the find command to identify files and piping
the results to rm was much faster than using rm -r.
Also, mounting the fs with the nodiratime option and setting:
<dlm plock_ownership="1" plock_rate_limit="0"/>
<gfs_controld plock_rate_limit="0"/>
in cluster.conf reduced rm -r time on the directory referenced below
from ~ 9 mins to ~ 5 mins.
Since we now understand the problem a little bit better we are able to
implement some workarounds to get us by.
Peter
~
On Wed, Jul 8, 2009 at 1:58 PM, Peter Schobel<pschobel@xxxxxxxxxx> wrote:
> I am trying to set up a four node cluster but am getting very poor
> performance when removing large directories. A directory approximately
> 1.6G in size takes around 5 mins to remove from the gfs2 filesystem
> but removes in around 10 seconds from the local disk.
>
> I am using CentOS 5.3 with kernel 2.6.18-128.1.16.el5PAE.
>
> The filesystem was formatted in the following manner: mkfs.gfs2 -t
> wtl_build:dev_home00 -p lock_dlm -j 10
> /dev/mapper/VolGroupGFS-LogVolDevHome00 and is being mounted with the
> following options: _netdev,noatime,defaults.
>
> If anyone knows what could be causing this please let me know. I'm
> happy to provide any other information.
>
> Regards,
>
> --
> Peter Schobel
> ~
>
--
Peter Schobel
~
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
