Thank you for the acknowledgement.
On Thu, Aug 31, 2017 at 8:30 PM, mohammad kashif <kashif.alig@xxxxxxxxx> wrote:
KashifCheersIt may be due to high load that status command failed. I will keep an eye and see whether it happens again.Hi AtinThanks, I was not running any script or gluster command. But now gluster status command started working. CPU usage also came down and looking at the ganglia graph, cpu usage is strongly correlated with network activity.On Thu, Aug 31, 2017 at 2:40 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:On Wed, 30 Aug 2017 at 20:55, mohammad kashif <kashif.alig@xxxxxxxxx> wrote:When I tried from other node then it seems two nodes have Locking issueHiI am running a 400TB five node purely distributed gluster setup. I am troubleshooting an issue where some times files creation fails. I found that volume status is not working
gluster volume status
Another transaction is in progress for atlasglust. Please try again after sometime.
gluster volume status
Locking failed on pplxgluster01... Please check log file for details.
Locking failed on pplxgluster04... Please check log file for details.This suggests that there are concurrent gluster cli operations been performed on the same volume. Are you monitoring the cluster through nagios or you have a script on all the nodes which checks for volume's health in a period of interval? Please note glusterd will process one cli operation on a volume at one time, the rest all transactions on the same volume will be failed.______________________________Can you please suggest that how to troubleshoot further?Gluster version is 3.11.2-1Also noticed that glusterfsd process is using around 1000% cpu usage. It is a decent server with 16 core and 64GB RAM.ThanksKashif
_________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users --- Atin (atinm)
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
