Hopefully this will help some people... try disabling the io-cache routine in the fuse configurations for your share. Let me know if you need instruction on doing this. It solved all of the lockup issues I was experiencing. I believe there is some sort of as-yet-undetermined memory leak here. Justice London -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of bxmatus at gmail.com Sent: Wednesday, June 08, 2011 12:22 PM To: gluster-users at gluster.org Subject: Re: uninterruptible processes writing toglusterfsshare Im using kernel 2.6.34 + fuse 2.5.5 + gluster 3.2 from beginning and it happen again today ... php-fpm freeze and reboot was only solution. Matus 2011/6/7 Markus Fr?hlich <markus.froehlich at xidras.com>: > hi! > > there ist no relavant output from dmesg. > no entries in the server log - only the one line in the client-server log, I > already posted. > > the glusterfs version on the server had been updated to gfs 3.2.0 more than > a month ago. > because of the troubles on the backup server, I deleted the whole backup > share and started from scratch. > > > I looked for a update of "fuse" and upgraded from 2.7.2-61.18.1 to > 2.8.5-41.1 > maybe this helps. > > here is the changelog info: > > Authors: > -------- > ? ?Miklos Szeredi <miklos at szeredi.hu> > Distribution: systemsmanagement:baracus / SLE_11_SP1 > * Tue Mar 29 2011 dbahi at novell.com > - remove the --no-canonicalize usage for suse_version <= 11.3 > > * Mon Mar 21 2011 coolo at novell.com > - licenses package is about to die > > * Thu Feb 17 2011 mszeredi at suse.cz > - In case of failure to add to /etc/mtab don't umount. [bnc#668820] > ?[CVE-2011-0541] > > * Tue Nov 16 2010 mszeredi at suse.cz > - Fix symlink attack for mount and umount [bnc#651598] > > * Wed Oct 27 2010 mszeredi at suse.cz > - Remove /etc/init.d/boot.fuse [bnc#648843] > > * Tue Sep 28 2010 mszeredi at suse.cz > - update to 2.8.5 > ?* fix option escaping for fusermount [bnc#641480] > > * Wed Apr 28 2010 mszeredi at suse.cz > - keep examples and internal docs in devel package (from jnweiger) > > * Mon Apr 26 2010 mszeredi at suse.cz > - update to 2.8.4 > ?* fix checking for symlinks in umount from /tmp > ?* fix umounting if /tmp is a symlink > > > kind regards > markus froehlich > > Am 06.06.2011 21:19, schrieb Anthony J. Biacco: >> >> Could be fuse, check 'dmesg' for kernel module timeouts. >> >> In a similar vein, has anyone seen signifigant performance/reliability >> with diff fuse versions? say, latest source vs. Rhel distro rpms vers. >> >> -Tony >> >> >> >> -----Original Message----- >> From: Mohit Anchlia<mohitanchlia at gmail.com> >> Sent: June 06, 2011 1:14 PM >> To: Markus Fr?hlich<markus.froehlich at xidras.com> >> Cc: gluster-users at gluster.org<gluster-users at gluster.org> >> Subject: Re: uninterruptible processes writing to >> glusterfsshare >> >> Is there anything in the server logs? Does it follow any particular >> pattern before going in this mode? >> >> Did you upgrade Gluster or is this new install? >> >> 2011/6/6 Markus Fr?hlich<markus.froehlich at xidras.com>: >>> >>> hi! >>> >>> sometimes we've on some client-servers hanging uninterruptible processes >>> ("ps aux" stat is on "D" ) and on one the CPU wait I/O grows within some >>> minutes to 100%. >>> you are not able to kill such processes - also "kill -9" doesnt work - >>> when >>> you connect via "strace" to such an process, you wont see anything and >>> you >>> cannot detach it again. >>> >>> there are only two possibilities: >>> killing the glusterfs process (umount GFS share) or rebooting the server. >>> >>> the only log entry I found, was on one client - just a single line: >>> [2011-06-06 10:44:18.593211] I >>> [afr-common.c:581:afr_lookup_collect_xattr] >>> 0-office-data-replicate-0: data self-heal is pending for >>> >>> /pc-partnerbet-public/Promotionaktionen/Mailakquise_2009/Webmaster_2010/HTML /bilder/Thumbs.db. >>> >>> one of the client-servers is a samba-server, the other one a >>> backup-server >>> based on rsync with millions of small files. >>> >>> gfs-servers + gfs-clients: SLES11 x86_64, glusterfs V 3.2.0 >>> >>> and here are the configs from server and client: >>> server config >>> >>> "/etc/glusterd/vols/office-data/office-data.gfs-01-01.GFS-office-data02.vol" : >>> volume office-data-posix >>> ? ?type storage/posix >>> ? ?option directory /GFS/office-data02 >>> end-volume >>> >>> volume office-data-access-control >>> ? ?type features/access-control >>> ? ?subvolumes office-data-posix >>> end-volume >>> >>> volume office-data-locks >>> ? ?type features/locks >>> ? ?subvolumes office-data-access-control >>> end-volume >>> >>> volume office-data-io-threads >>> ? ?type performance/io-threads >>> ? ?subvolumes office-data-locks >>> end-volume >>> >>> volume office-data-marker >>> ? ?type features/marker >>> ? ?option volume-uuid 3c6e633d-a0bb-4c52-8f05-a2db9bc9c659 >>> ? ?option timestamp-file /etc/glusterd/vols/office-data/marker.tstamp >>> ? ?option xtime off >>> ? ?option quota off >>> ? ?subvolumes office-data-io-threads >>> end-volume >>> >>> volume /GFS/office-data02 >>> ? ?type debug/io-stats >>> ? ?option latency-measurement off >>> ? ?option count-fop-hits off >>> ? ?subvolumes office-data-marker >>> end-volume >>> >>> volume office-data-server >>> ? ?type protocol/server >>> ? ?option transport-type tcp >>> ? ?option auth.addr./GFS/office-data02.allow * >>> ? ?subvolumes /GFS/office-data02 >>> end-volume >>> >>> >>> -------------- >>> client config "/etc/glusterd/vols/office-data/office-data-fuse.vol": >>> volume office-data-client-0 >>> ? ?type protocol/client >>> ? ?option remote-host gfs-01-01 >>> ? ?option remote-subvolume /GFS/office-data02 >>> ? ?option transport-type tcp >>> end-volume >>> >>> volume office-data-replicate-0 >>> ? ?type cluster/replicate >>> ? ?subvolumes office-data-client-0 >>> end-volume >>> >>> volume office-data-write-behind >>> ? ?type performance/write-behind >>> ? ?subvolumes office-data-replicate-0 >>> end-volume >>> >>> volume office-data-read-ahead >>> ? ?type performance/read-ahead >>> ? ?subvolumes office-data-write-behind >>> end-volume >>> >>> volume office-data-io-cache >>> ? ?type performance/io-cache >>> ? ?subvolumes office-data-read-ahead >>> end-volume >>> >>> volume office-data-quick-read >>> ? ?type performance/quick-read >>> ? ?subvolumes office-data-io-cache >>> end-volume >>> >>> volume office-data-stat-prefetch >>> ? ?type performance/stat-prefetch >>> ? ?subvolumes office-data-quick-read >>> end-volume >>> >>> volume office-data >>> ? ?type debug/io-stats >>> ? ?option latency-measurement off >>> ? ?option count-fop-hits off >>> ? ?subvolumes office-data-stat-prefetch >>> end-volume >>> >>> >>> ?-- Mit freundlichen Gr?ssen >>> >>> Markus Fr?hlich >>> Techniker >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
