A hard reboot solved it for me too, but I am a little worried on the stability of it.
The entire wordpress web root is stored on the gluster volume, a day later I attempted to tar/gzip the web directory on the gluster volume and again the load on one particular server sky rocketed, server became unresponsive and kernel hang messages appeared claiming the gzip process was hanging. It seems that if I cause any serious IO on the volume, things seem to snowball.
There are 3 web nodes with a replicated gluster volume between them all. Each web node also mounts the gluster volume locally using NFS (an attempt to improve performance for lots of small php files). I tried mounting native glusterfs instead of NFS for client mount, but I had the same issues.
ii glusterfs-client 3.6.2-ubuntu1~trusty3 amd64 clustered file-system (client package)
ii glusterfs-common 3.6.2-ubuntu1~trusty3 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.6.2-ubuntu1~trusty3 amd64 clustered file-system (server package)
Volume Name: my_filestore_vol
Type: Replicate
Volume ID: xyz
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: my-web01:/export/brick0
Brick2: my-web02:/export/brick0
Brick3: my-web03:/export/brick0
Options Reconfigured:
nfs.drc: off
diagnostics.brick-log-level: WARNING
Status of volume: my_filestore_vol
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick my-web01:/export/brick0 49152 Y 2138
Brick my-web02:/export/brick0 49152 Y 21104
Brick my-web03:/export/brick0 49152 Y 1827
NFS Server on localhost 2049 Y 2145
Self-heal Daemon on localhost N/A Y 2152
NFS Server on my-web03 2049 Y 1834
Self-heal Daemon on my-web03 N/A Y 1841
NFS Server on my-web02 2049 Y 21118
Self-heal Daemon on my-web02 N/A Y 21123
Task Status of Volume my_filestore_vol
------------------------------------------------------------------------------
There are no active volume tasks
Regards,
A
From: "Hoggins!" <fuckspam@xxxxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Sent: Friday, 8 May, 2015 1:25:26 PM
Subject: Re: High load / hang
Well, that's "funny", because the exact same thing happened to me this morning, except that I could hard reboot the machine, and it got up and running normally again.
But the symptoms you describe are oddly similar, and strangely simultaneous.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
To: gluster-users@xxxxxxxxxxx
Sent: Friday, 8 May, 2015 1:25:26 PM
Subject: Re: High load / hang
Well, that's "funny", because the exact same thing happened to me this morning, except that I could hard reboot the machine, and it got up and running normally again.
But the symptoms you describe are oddly similar, and strangely simultaneous.
Le 08/05/2015 10:36, Alun James a
écrit :
Hi folks,
I have a 3 node gluster/web/db cluster running a Wordpress site . This morning one of the nodes is under very high load and the mounted gluster partition is inaccessible. Attempts to reboot that node have failed with the server seemingly hung/blocked. Has anyone else experienced this or can give any pointers in how to diagnose the cause of gluster going awry?
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
