On 04/23/2015 08:05 PM, free.aaa wrote: > Hi everybody! > > I have "gluster peer probe gfs1" command hung with the result of "Probe > Sent to Peer (connected)" >> gfs3#gluster peer status >> Number of Peers: 3 >> >> Hostname: gfs6 >> Uuid: 6bd6ee25-e257-4703-b500-330741b90471 >> State: Peer in Cluster (Connected) >> >> Hostname: gfs4 >> Uuid: bb1bed20-25bf-43b0-8faa-49f1b5b9ae59 >> State: Peer in Cluster (Connected) >> >> Hostname: gfs1 >> Uuid: bb67c1da-2698-4c35-b29d-f80f8eb814a6 >> State: Probe Sent to Peer (Connected) > > I double checked the dns resolution. Forward and reverse resolution > works fine. > I tried to debug on gfs1 and gfs3 at the moment of probe command and it > seems to me that gfs1 does not sync all the gluster configuration from > gfs3. Especially peers folder contains only 1 peer with the following > content: >> gfs1#cat /var/lib/glusterd/peers/192.168.9.53 >> uuid=00000000-0000-0000-0000-000000000000 >> state=8 >> hostname1=192.168.9.53 > so there are no uuids both in the content of the file and in filename > itself. Besides there no info about other peers like gfs4, gfs6 in the > folder. gfs1 would not have all the peers configuration until and unless it becomes a node in the cluster. Here the peer handshaking between gfs1 & gfs3 did not happen completely, some how gfs3 didn't receive a response which it was expecting from gfs1. Can you check whether glusterd instance of gfs1 some how went down at the point when peer probe was executed? > Last lines of the debug log on gfs1 looks like this: >> [2015-04-23 10:12:28.334239] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.54 >> [2015-04-23 10:12:28.334413] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.54 >> [2015-04-23 10:12:28.334557] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.54 >> [2015-04-23 10:12:28.334697] D [common-utils.c:2946:gf_is_local_addr] >> 0-management: gfs4 is not local >> [2015-04-23 10:12:28.334707] D >> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: >> returning -1 >> [2015-04-23 10:12:28.334714] D >> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: >> Returning -1 >> [2015-04-23 10:12:28.336881] D >> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management: >> Unable to find friend: gfs6 >> [2015-04-23 10:12:28.337490] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.56 >> [2015-04-23 10:12:28.337697] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.56 >> [2015-04-23 10:12:28.337841] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.56 >> [2015-04-23 10:12:28.337981] D [common-utils.c:2946:gf_is_local_addr] >> 0-management: gfs6 is not local >> [2015-04-23 10:12:28.337991] D >> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: >> returning -1 >> [2015-04-23 10:12:28.337998] D >> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: >> Returning -1 >> [2015-04-23 10:12:28.338858] D >> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management: >> Friend gfs3 found.. state: 8 >> [2015-04-23 10:12:28.338873] D >> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: >> returning 0 >> [2015-04-23 10:12:28.340989] D >> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management: >> Unable to find friend: gfs6 >> [2015-04-23 10:12:28.341545] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.56 >> [2015-04-23 10:12:28.341697] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.56 >> [2015-04-23 10:12:28.341840] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.56 >> [2015-04-23 10:12:28.341980] D [common-utils.c:2946:gf_is_local_addr] >> 0-management: gfs6 is not local >> [2015-04-23 10:12:28.341991] D >> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: >> returning -1 >> [2015-04-23 10:12:28.341997] D >> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1 >> [2015-04-23 10:12:28.342003] D >> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: >> Returning -1 >> [2015-04-23 10:12:28.344065] D >> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management: >> Unable to find friend: gfs4 >> [2015-04-23 10:12:28.344620] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.54 >> [2015-04-23 10:12:28.344772] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.54 >> [2015-04-23 10:12:28.344914] D [common-utils.c:2930:gf_is_local_addr] >> 0-management: 192.168.9.54 >> [2015-04-23 10:12:28.345054] D [common-utils.c:2946:gf_is_local_addr] >> 0-management: gfs4 is not local >> [2015-04-23 10:12:28.345064] D >> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: >> returning -1 >> [2015-04-23 10:12:28.345071] D >> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: >> Returning -1 >> [2015-04-23 10:12:28.345543] D [run.c:190:runner_log] 0-: Starting the >> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id >> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l >> /var/log/glusterfs/quotad.log -S >> /var/run/3e619fbfe69c96b1dbc7486a7d38a7be.socket --xlator-option >> *replicate*.data-self-heal=off --xlator-option >> *replicate*.metadata-self-heal=off --xlator-option >> *replicate*.entry-self-heal=off >> ^C[2015-04-23 10:20:50.237403] W [glusterfsd.c:1095:cleanup_and_exit] >> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_end_reuse+0x26) >> [0x7fa81cd3dfb6] >> (-->/lib/x86_64-linux-gnu/libpthread.so.0(waitpid+0x5b) >> [0x7fa81c499c8b] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x321e0) >> [0x7fa81bd3a1e0]))) 0-: received signum (2), shutting down >> [2015-04-23 10:20:50.237436] D >> [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: >> portmapper signout arguments not given > So I think server hangs when trying to start volume with references to > gfs4 and gfs6 which it does not know about, but why gfs1 peers > configuration is empty? > > Any help? > > Thanks in advance! > Best regards, > Alex. > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users > > -- ~Atin _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
