|
responses below Ted Miller On 5/26/2015 12:01 AM, Atin Mukherjee
wrote:
attached as etc-glusterfs-glusterd.vol.log.newer.2, starting from last machine rebootOn 05/26/2015 03:12 AM, Ted Miller wrote:From: Niels de Vos <ndevos@xxxxxxxxxx> Sent: Monday, May 25, 2015 4:44 PM On Mon, May 25, 2015 at 06:49:26PM +0000, Ted Miller wrote:________________________________ From: Humble Devassy Chirammal <humble.devassy@xxxxxxxxx> Sent: Monday, May 18, 2015 9:37 AM Hi All, GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are available at download.gluster.org<http://download.gluster.org> [1]. [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/ --Humble On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur <vbellur@xxxxxxxxxx<mailto:vbellur@xxxxxxxxxx>> wrote: Hi All, I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 contains several [snip] Cheers, Vijay [snip] What happened to packages for RHEL/Centos 5? I have the (probably unusual--added gluster to existing servers) setup of running a replica 3 cluster where two nodes run on Centos 6 and one is still on Centos 5. This is a personal setup, and I have been using http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml as my repo. It has worked fine for a while, but this time the two Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind at 3.6.3.Packages for RHEL/CentOS-5 are not available yet. These will follow later. Thare are some changes needed to be able to build the packages on EL5. Because we are currently stabilizing our CI/regression tests, we do not merge any other changes. Until we provide packages in our repository, you could apply patch http://review.gluster.org/10803 yourself and build the EL5 version. I expect that we will do a release in 2-3 weeks which will have EL5 RPMs too. I have no idea about the problem below, it sounds like something the GlusterD developers could help with. NielsCommand 'gluster volume status' on the C5 machine makes everything look fine: Status of volume: ISO2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.x.x.2:/bricks/01/iso2 49162 Y 4679 Brick 10.x.x.4:/bricks/01/iso2 49183 Y 6447 Brick 10.x.x.9:/bricks/01/iso2 49169 Y 1985 But the same command on either of the C6 machines shows the C5 machine (10.x.x.2) missing in action (though it does recognize that there are NFS and heal daemons there): Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 NFS Server on localhost 2049 0 Y 2279 Self-heal Daemon on localhost N/A N/A Y 2754 NFS Server on 10.41.65.2 2049 0 Y 4757 Self-heal Daemon on 10.41.65.2 N/A N/A Y 4764 NFS Server on 10.41.65.4 2049 0 Y 6543 Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551 So, is this just an oversight (I hope), or has support for C5 been dropped? If support for C5 is gone, how do I downgrade my Centos6 machines back to 3.6.x? (I know how to change the repo, but the actual sequence of yum commands and gluster commands is unknown to me).Could you attach the glusterd log file of 10.x.x.2 machine attached as etc-glusterfs-glusterd.vol.log.newer4 starting same time as .2 logand the node from where you triggered volume status. I have several volumes, so I chose the one that shows up first on the listings:Could you also share gluster volume info output of all the nodes? from 10.41.65.2: [root@office2 /var/log/glusterfs]$ gluster volume info Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root@office2 /var/log/glusterfs]$ gluster volume status ISO2 Status of volume: ISO2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.2:/bricks/01/iso2 49162 Y 4463 Brick 10.41.65.4:/bricks/01/iso2 49183 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 Y 1985 NFS Server on localhost 2049 Y 4536 Self-heal Daemon on localhost N/A Y 4543 NFS Server on 10.41.65.9 2049 Y 2279 Self-heal Daemon on 10.41.65.9 N/A Y 2754 NFS Server on 10.41.65.4 2049 Y 6543 Self-heal Daemon on 10.41.65.4 N/A Y 6551 Task Status of Volume ISO2 ------------------------------------------------------------------------------ There are no active volume tasks [root@office2 ~]$ gluster peer status Number of Peers: 2 Hostname: 10.41.65.9 Uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 State: Peer in Cluster (Connected) Hostname: 10.41.65.4 Uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c State: Peer in Cluster (Connected) from 10.41.65.4: [root@office4b ~]# gluster volume info ISO2 Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root@office4b ~]# gluster volume status ISO2 Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 NFS Server on localhost 2049 0 Y 6543 Self-heal Daemon on localhost N/A N/A Y 6551 NFS Server on 10.41.65.2 2049 0 Y 4536 Self-heal Daemon on 10.41.65.2 N/A N/A Y 4543 NFS Server on 10.41.65.9 2049 0 Y 2279 Self-heal Daemon on 10.41.65.9 N/A N/A Y 2754 Task Status of Volume ISO2 ------------------------------------------------------------------------------ There are no active volume tasks [root@office4b ~]# gluster peer status Number of Peers: 2 Hostname: 10.41.65.2 Uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 State: Peer in Cluster (Connected) Hostname: 10.41.65.9 Uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 State: Peer in Cluster (Connected) from 10.41.65.9: [root@office9 ~]$ gluster volume info ISO2 Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root@office9 ~]$ gluster volume status ISO2 Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 NFS Server on localhost 2049 0 Y 2279 Self-heal Daemon on localhost N/A N/A Y 2754 NFS Server on 10.41.65.2 2049 0 Y 4536 Self-heal Daemon on 10.41.65.2 N/A N/A Y 4543 NFS Server on 10.41.65.4 2049 0 Y 6543 Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551 Task Status of Volume ISO2 ------------------------------------------------------------------------------ There are no active volume tasks [root@office9 ~]$ gluster peer status Number of Peers: 2 Hostname: 10.41.65.2 Uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 State: Peer in Cluster (Connected) Hostname: 10.41.65.4 Uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c State: Peer in Cluster (Connected) Ted Miller Elkhart, IN, USAThanks for the information. As long as I know it is coming, I can improvise and hang on. I am assuming that the problem with the .2 machine not being seen is a result of running a cluster with a version split. Ted Miller _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users --
Ted Miller, Design Engineer SonSet Solutions (formerly HCJB Global Technology Center) my desk +1 574.970.4272 receptionist +1 574.972.4252 http://sonsetsolutions.org Technology for abundant life! |
[2015-05-25 14:51:53.898279] W [socket.c:620:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (No data available) [2015-05-25 14:51:53.898403] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer cf2ae9c7-833e-4a73-a996-e72158011c69, in Peer in Cluster state, has disconnected from glusterd. [2015-05-25 14:51:53.898800] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:51:53.899083] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:51:53.899356] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:51:53.899647] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:51:53.899958] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:51:53.900230] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:51:53.900498] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:51:53.900767] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:51:53.901034] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:51:53.901304] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:51:53.901591] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:51:53.901861] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:03.883612] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:04.100889] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:04.153955] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:04.214125] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:04.214210] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:04.253289] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:05.876100] W [socket.c:620:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (Connection reset by peer) [2015-05-25 14:52:05.876432] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d7)[0x315680e887] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x315680e98e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x315680ea31] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd2)[0x315680f152] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2015-05-25 14:52:04.163362 (xid=0xe) [2015-05-25 14:52:05.876479] E [glusterd-rpc-ops.c:619:__glusterd_friend_update_cbk] 0-management: RPC Error [2015-05-25 14:52:05.876515] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000 [2015-05-25 14:52:05.876706] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d7)[0x315680e887] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x315680e98e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x315680ea31] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd2)[0x315680f152] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2015-05-25 14:52:04.166562 (xid=0xf) [2015-05-25 14:52:05.876751] E [glusterd-handshake.c:1680:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later [2015-05-25 14:52:05.877044] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:52:05.877336] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:52:05.877610] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:52:05.877879] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:52:05.878149] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:52:05.878418] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:52:05.878686] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:52:05.878955] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:52:05.879235] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:52:05.879538] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:52:05.880094] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:52:05.880366] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:08.846394] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:10.144090] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.156233] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:10.161604] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:10.161697] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.161767] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.474292] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69, host: 10.41.65.9, port: 0 [2015-05-25 14:52:10.528558] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.528639] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.813342] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:10.813441] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.997945] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:05.876801] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer cf2ae9c7-833e-4a73-a996-e72158011c69, in Peer in Cluster state, has disconnected from glusterd.
[2015-05-25 14:51:53.902180] W [socket.c:642:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (No data available) [2015-05-25 14:51:53.960843] I [MSGID: 106004] [glusterd-handler.c:4809:__glusterd_peer_rpc_notify] 0-management: Peer <10.41.65.9> (<cf2ae9c7-833e-4a73-a996-e72158011c69>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-05-25 14:51:53.963643] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:51:53.963793] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:51:53.963927] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:51:53.964057] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:51:53.964187] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:51:53.964316] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:51:53.964452] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:51:53.964587] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:51:53.964716] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:51:53.964845] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:51:53.964974] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:51:53.965103] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:03.879790] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:04.097083] I [glusterd-handler.c:2422:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:04.161852] I [glusterd-handler.c:3580:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:04.249180] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:04.249236] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:04.249279] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:05.871153] W [socket.c:642:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (Connection reset by peer) [2015-05-25 14:52:05.871411] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f066ce7fa87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f066ce7fb9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f066ce7fc6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f066ce8022f] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2015-05-25 14:52:04.210183 (xid=0xa) [2015-05-25 14:52:05.871431] E [glusterd-rpc-ops.c:653:__glusterd_friend_update_cbk] 0-management: RPC Error [2015-05-25 14:52:05.871441] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000 [2015-05-25 14:52:05.871525] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f066ce7fa87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f066ce7fb9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f066ce7fc6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f066ce8022f] ))))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2015-05-25 14:52:04.249148 (xid=0xb) [2015-05-25 14:52:05.871545] E [glusterd-handshake.c:1740:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later [2015-05-25 14:52:05.871564] I [MSGID: 106004] [glusterd-handler.c:4809:__glusterd_peer_rpc_notify] 0-management: Peer <10.41.65.9> (<cf2ae9c7-833e-4a73-a996-e72158011c69>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-05-25 14:52:05.871701] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:52:05.871832] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:52:05.871962] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:52:05.872091] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:52:05.872248] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:52:05.872385] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:52:05.872517] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:52:05.872646] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:52:05.872774] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:52:05.872903] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:52:05.873032] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:52:05.873167] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:08.842559] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:09.710802] I [glusterd-rpc-ops.c:463:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69, host: 10.41.65.9, port: 0 [2015-05-25 14:52:10.155216] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:10.155267] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.737330] I [glusterd-handler.c:2422:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.766113] I [glusterd-handler.c:3580:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:10.852943] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:10.852988] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.853013] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.853061] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.853079] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:11.047970] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
