Gluster failure due to "0-management: Lock not released for <volumename>"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi All,

 

I’m fairly new to Gluster (3.10.3) and got it going for a couple of months now but suddenly after a power failure in our building it all came crashing down.  No client is able to connect after powering back the 3 nodes I have setup.

 

Looking at the logs, it looks like there’s some sort of “Lock” placed on the volume which prevents all the clients from connecting to the Gluster endpoint.

 

I can’t even do a #gluster volume status all command IF more than 1 node is powered up.  I have to shutdown node2-3 and then I am able to issue the command on node1 to see volume status.  When all nodes are powered up and I check the peer status, it says that all peers are connected.  Trying to connect to the Gluster volume from all clients says gluster endpoint is not available and times out. There are no network issues and each node can ping each other and there are no firewalls or any other device between the nodes and clients.

 

Please help if you think you know how to fix this.  I have a feeling it’s this “lock” that’s not “released” due to the whole setup losing power all of a sudden.  I’ve tried restarting all the nodes, restarting glusterfs-server etc. I’m out of ideas.

 

Thanks in advance!

 

Victor

 

Volume Name: teravolume

Type: Distributed-Replicate

Volume ID: 85af74d0-f1bc-4b0d-8901-4dea6e4efae5

Status: Started

Snapshot Count: 0

Number of Bricks: 3 x 2 = 6

Transport-type: tcp

Bricks:

Brick1: gfsnode1:/media/brick1

Brick2: gfsnode2:/media/brick1

Brick3: gfsnode3:/media/brick1

Brick4: gfsnode1:/media/brick2

Brick5: gfsnode2:/media/brick2

Brick6: gfsnode3:/media/brick2

Options Reconfigured:

nfs.disable: on

 

 

[2017-06-21 16:02:52.376709] W [MSGID: 106118] [glusterd-handler.c:5913:__glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume

[2017-06-21 16:03:03.429032] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000

[2017-06-21 16:13:13.326478] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x105 sent = 2017-06-21 16:03:03.202284. timeout = 600 for 192.168.150.52:$

[2017-06-21 16:13:13.326519] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x105 sent = 2017-06-21 16:03:03.204555. timeout = 600 for 192.168.150.53:$

[2017-06-21 16:18:34.456522] I [MSGID: 106004] [glusterd-handler.c:5888:__glusterd_peer_rpc_notify] 0-management: Peer <gfsnode2> (<e1e1caa5-9842-40d8-8492-a82b079879a3>), in state <Peer in Cluste$

[2017-06-21 16:18:34.456619] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x1f879) [0x7fee6bc22879] -->/usr/lib/x86_64-l$

[2017-06-21 16:18:34.456638] W [MSGID: 106118] [glusterd-handler.c:5913:__glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume

[2017-06-21 16:18:34.456661] I [MSGID: 106004] [glusterd-handler.c:5888:__glusterd_peer_rpc_notify] 0-management: Peer <gfsnode3> (<59b9effa-2b88-4764-9130-4f31c14c362e>), in state <Peer in Cluste$

[2017-06-21 16:18:34.456692] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x1f879) [0x7fee6bc22879] -->/usr/lib/x86_64-l$

[2017-06-21 16:18:43.323944] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000

[2017-06-21 16:18:34.456699] W [MSGID: 106118] [glusterd-handler.c:5913:__glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume

[2017-06-21 16:18:45.628552] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000

[2017-06-21 16:23:40.607173] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume teravolume

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux