I have been working with the gluster support department (under paid support I should mention)... with no resolution or similar. A bug report specifically was not filed by myself though. Justice London -----Original Message----- From: Mohit Anchlia [mailto:mohitanchlia at gmail.com] Sent: Wednesday, June 08, 2011 12:29 PM To: Justice London Cc: bxmatus at gmail.com; gluster-users at gluster.org Subject: Re: uninterruptible processes writing toglusterfsshare On Wed, Jun 8, 2011 at 12:29 PM, Justice London <jlondon at lawinfo.com> wrote: > Hopefully this will help some people... try disabling the io-cache routine > in the fuse configurations for your share. Let me know if you need > instruction on doing this. It solved all of the lockup issues I was > experiencing. I believe there is some sort of as-yet-undetermined memory > leak here. Was there a bug filed? If you think this is a bug it will help others as well. > > Justice London > > -----Original Message----- > From: gluster-users-bounces at gluster.org > [mailto:gluster-users-bounces at gluster.org] On Behalf Of bxmatus at gmail.com > Sent: Wednesday, June 08, 2011 12:22 PM > To: gluster-users at gluster.org > Subject: Re: uninterruptible processes writing > toglusterfsshare > > Im using kernel 2.6.34 + fuse 2.5.5 + gluster 3.2 from beginning and > it happen again today ... > php-fpm freeze and reboot was only solution. > > Matus > > > 2011/6/7 Markus Fr?hlich <markus.froehlich at xidras.com>: >> hi! >> >> there ist no relavant output from dmesg. >> no entries in the server log - only the one line in the client-server log, > I >> already posted. >> >> the glusterfs version on the server had been updated to gfs 3.2.0 more > than >> a month ago. >> because of the troubles on the backup server, I deleted the whole backup >> share and started from scratch. >> >> >> I looked for a update of "fuse" and upgraded from 2.7.2-61.18.1 to >> 2.8.5-41.1 >> maybe this helps. >> >> here is the changelog info: >> >> Authors: >> -------- >> ? ?Miklos Szeredi <miklos at szeredi.hu> >> Distribution: systemsmanagement:baracus / SLE_11_SP1 >> * Tue Mar 29 2011 dbahi at novell.com >> - remove the --no-canonicalize usage for suse_version <= 11.3 >> >> * Mon Mar 21 2011 coolo at novell.com >> - licenses package is about to die >> >> * Thu Feb 17 2011 mszeredi at suse.cz >> - In case of failure to add to /etc/mtab don't umount. [bnc#668820] >> ?[CVE-2011-0541] >> >> * Tue Nov 16 2010 mszeredi at suse.cz >> - Fix symlink attack for mount and umount [bnc#651598] >> >> * Wed Oct 27 2010 mszeredi at suse.cz >> - Remove /etc/init.d/boot.fuse [bnc#648843] >> >> * Tue Sep 28 2010 mszeredi at suse.cz >> - update to 2.8.5 >> ?* fix option escaping for fusermount [bnc#641480] >> >> * Wed Apr 28 2010 mszeredi at suse.cz >> - keep examples and internal docs in devel package (from jnweiger) >> >> * Mon Apr 26 2010 mszeredi at suse.cz >> - update to 2.8.4 >> ?* fix checking for symlinks in umount from /tmp >> ?* fix umounting if /tmp is a symlink >> >> >> kind regards >> markus froehlich >> >> Am 06.06.2011 21:19, schrieb Anthony J. Biacco: >>> >>> Could be fuse, check 'dmesg' for kernel module timeouts. >>> >>> In a similar vein, has anyone seen signifigant performance/reliability >>> with diff fuse versions? say, latest source vs. Rhel distro rpms vers. >>> >>> -Tony >>> >>> >>> >>> -----Original Message----- >>> From: Mohit Anchlia<mohitanchlia at gmail.com> >>> Sent: June 06, 2011 1:14 PM >>> To: Markus Fr?hlich<markus.froehlich at xidras.com> >>> Cc: gluster-users at gluster.org<gluster-users at gluster.org> >>> Subject: Re: uninterruptible processes writing to >>> glusterfsshare >>> >>> Is there anything in the server logs? Does it follow any particular >>> pattern before going in this mode? >>> >>> Did you upgrade Gluster or is this new install? >>> >>> 2011/6/6 Markus Fr?hlich<markus.froehlich at xidras.com>: >>>> >>>> hi! >>>> >>>> sometimes we've on some client-servers hanging uninterruptible processes >>>> ("ps aux" stat is on "D" ) and on one the CPU wait I/O grows within some >>>> minutes to 100%. >>>> you are not able to kill such processes - also "kill -9" doesnt work - >>>> when >>>> you connect via "strace" to such an process, you wont see anything and >>>> you >>>> cannot detach it again. >>>> >>>> there are only two possibilities: >>>> killing the glusterfs process (umount GFS share) or rebooting the > server. >>>> >>>> the only log entry I found, was on one client - just a single line: >>>> [2011-06-06 10:44:18.593211] I >>>> [afr-common.c:581:afr_lookup_collect_xattr] >>>> 0-office-data-replicate-0: data self-heal is pending for >>>> >>>> > /pc-partnerbet-public/Promotionaktionen/Mailakquise_2009/Webmaster_2010/HTML > /bilder/Thumbs.db. >>>> >>>> one of the client-servers is a samba-server, the other one a >>>> backup-server >>>> based on rsync with millions of small files. >>>> >>>> gfs-servers + gfs-clients: SLES11 x86_64, glusterfs V 3.2.0 >>>> >>>> and here are the configs from server and client: >>>> server config >>>> >>>> > "/etc/glusterd/vols/office-data/office-data.gfs-01-01.GFS-office-data02.vol" > : >>>> volume office-data-posix >>>> ? ?type storage/posix >>>> ? ?option directory /GFS/office-data02 >>>> end-volume >>>> >>>> volume office-data-access-control >>>> ? ?type features/access-control >>>> ? ?subvolumes office-data-posix >>>> end-volume >>>> >>>> volume office-data-locks >>>> ? ?type features/locks >>>> ? ?subvolumes office-data-access-control >>>> end-volume >>>> >>>> volume office-data-io-threads >>>> ? ?type performance/io-threads >>>> ? ?subvolumes office-data-locks >>>> end-volume >>>> >>>> volume office-data-marker >>>> ? ?type features/marker >>>> ? ?option volume-uuid 3c6e633d-a0bb-4c52-8f05-a2db9bc9c659 >>>> ? ?option timestamp-file /etc/glusterd/vols/office-data/marker.tstamp >>>> ? ?option xtime off >>>> ? ?option quota off >>>> ? ?subvolumes office-data-io-threads >>>> end-volume >>>> >>>> volume /GFS/office-data02 >>>> ? ?type debug/io-stats >>>> ? ?option latency-measurement off >>>> ? ?option count-fop-hits off >>>> ? ?subvolumes office-data-marker >>>> end-volume >>>> >>>> volume office-data-server >>>> ? ?type protocol/server >>>> ? ?option transport-type tcp >>>> ? ?option auth.addr./GFS/office-data02.allow * >>>> ? ?subvolumes /GFS/office-data02 >>>> end-volume >>>> >>>> >>>> -------------- >>>> client config "/etc/glusterd/vols/office-data/office-data-fuse.vol": >>>> volume office-data-client-0 >>>> ? ?type protocol/client >>>> ? ?option remote-host gfs-01-01 >>>> ? ?option remote-subvolume /GFS/office-data02 >>>> ? ?option transport-type tcp >>>> end-volume >>>> >>>> volume office-data-replicate-0 >>>> ? ?type cluster/replicate >>>> ? ?subvolumes office-data-client-0 >>>> end-volume >>>> >>>> volume office-data-write-behind >>>> ? ?type performance/write-behind >>>> ? ?subvolumes office-data-replicate-0 >>>> end-volume >>>> >>>> volume office-data-read-ahead >>>> ? ?type performance/read-ahead >>>> ? ?subvolumes office-data-write-behind >>>> end-volume >>>> >>>> volume office-data-io-cache >>>> ? ?type performance/io-cache >>>> ? ?subvolumes office-data-read-ahead >>>> end-volume >>>> >>>> volume office-data-quick-read >>>> ? ?type performance/quick-read >>>> ? ?subvolumes office-data-io-cache >>>> end-volume >>>> >>>> volume office-data-stat-prefetch >>>> ? ?type performance/stat-prefetch >>>> ? ?subvolumes office-data-quick-read >>>> end-volume >>>> >>>> volume office-data >>>> ? ?type debug/io-stats >>>> ? ?option latency-measurement off >>>> ? ?option count-fop-hits off >>>> ? ?subvolumes office-data-stat-prefetch >>>> end-volume >>>> >>>> >>>> ?-- Mit freundlichen Gr?ssen >>>> >>>> Markus Fr?hlich >>>> Techniker >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
