On 09.07.2011 09:41, Anand Avati wrote: > > > On Wed, Jul 6, 2011 at 6:58 AM, Tomasz Chmielewski <mangoo at wpkg.org > <mailto:mangoo at wpkg.org>> wrote: > > > One of the peers is behaving very slow - load is constantly around > 6-15, and generally all connected clients work very slow. > > As soon as I shut that peer down, clients are relatively fast. > > > What I noticed, that on one gluster peer, I only get around 100 > files opened by the gluster process: > > # lsof -n|grep -c gluster > 96 > > > On the slow peer, it's hundreds, tens of thousands, and growing > constantly: > > > # while true; do lsof -n|grep -c gluster; sleep 10; done > 169934 > 170181 > 170363 > 170655 > > > If I restart glusterd on that peer, it starts to grow again. > > > > On both peers, I have a big number of such entries: > > [2011-07-06 03:22:00.20873] W > [server-resolve.c:556:server___resolve] 0-gluster-data-server: pure > path resolution for /some/path/.../file (LOOKUP) > > > Where the path is a symbolic link to the gluster mount. > > > > Can you describe this a little more? What do you mean by those paths are > symbolic links to the gluster mount? They are supposed to be files > within the gluster mount. Do you have a gluster mount on that peer as > well? If so, does unmounting have any effect? I mean, glusterfs is mounted in /home/glusterfs, with real data in /home/gluster-data. /some/ is a symbolic link to /home/www; where we have some more symlinks, this time to somewhere inside the gluster mount point, /home/glusterfs. For example, with a path like: /some/storage_cache/tatuaze/cache/photos/ we have: 1) symlink: /some -> /home/www 2) another symlink /home/www/storage_cache -> /home/glusterfs/storage_cache So the real path without symlinks is this one: /home/glusterfs/storage_cache/tatuaze/cache/photos/ > This is strange. There is almost no reason how you can be seeing this > error unless the extended attributes were modified outside gluster. Can > you get a dump of those attributes on those files with 'getfattr -d -m . > -e hex /some/path/../file' and post it here? # getfattr -d -m . -e hex /home/gluster-data/www/storage_cache/tatuaze/cache/photos/tatuaz/4662/tatuaze-na-przedramieniu-5663_5.jpg getfattr: Removing leading '/' from absolute path names # file: home/gluster-data/www/storage_cache/tatuaze/cache/photos/tatuaz/4662/tatuaze-na-przedramieniu-5663_5.jpg trusted.afr.gluster-data-client-0=0x000000000000000000000000 trusted.afr.gluster-data-client-1=0x000000000000000000000000 trusted.gfid=0xc46429f760cf4a278ae9bc38eec597f9 Note that the issue is somehow gone; I've made "ls -lR" on both backends, -- Tomasz Chmielewski http://wpkg.org
