I've got a 2 peer replicated setup with 2 bricks on each system.
Starting sometime overnight, one system ground to a halt, to the point a
local login was not possible.
Restarted.
A couple hours later I tried to log in via SSH, too many open files. A
quick 'lsof" showed thousands of open files on glusterfs.
I just restarted the server again...and appears to be doing the same
thing. Addition of nearly 2500 files in 4 minutes, not 5 minutes after
rebooting.
for d in {1..1000};do a=`date`;b=`lsof|grep gluster|wc -l`;echo $a " "
$b;sleep 60;done
Tue Aug 2 07:56:37 EDT 2011 2924
Tue Aug 2 07:57:37 EDT 2011 3568
Tue Aug 2 07:58:38 EDT 2011 4154
Tue Aug 2 07:59:38 EDT 2011 4718
Tue Aug 2 08:00:38 EDT 2011 5376
It's peer is more what's expected:
for d in {1..1000};do a=`date`;b=`lsof|grep gluster|wc -l`;echo $a " "
$b;sleep 60;done
Tue Aug 2 07:57:18 EDT 2011 193
Tue Aug 2 07:58:18 EDT 2011 192
Tue Aug 2 07:59:18 EDT 2011 193
Tue Aug 2 08:00:19 EDT 2011 193
Tue Aug 2 08:01:19 EDT 2011 191
Its running 3.2.2
Any suggestions to what's happening?
Where to start to troubleshoot.
D.
--
Dave Bour
Senior Consultant
Desktop Solution Center
905.381.0077 x501
dcbour at desktopsolutioncenter.ca
Linked In: http://ca.linkedin.com/in/dcbour
Facebook: http://www.facebook.com/dcbour
Providing Complete IT Peace of Mind
