On 2016-04-06 21:20, Atin Mukherjee wrote:
On 04/07/2016 04:04 AM, Ernie Dunbar wrote:
On 2016-04-06 11:42, Ernie Dunbar wrote:
I've already successfully created a Gluster cluster, but when I try
to
add a new node, gluster on the new node claims it can't find the
hostname of the first node in the cluster.
I've added the hostname nfs1.lightspeed.ca to /etc/hosts like this:
root@nfs3:/home/ernied# cat /etc/hosts
127.0.0.1 localhost
192.168.1.31 nfs1.lightspeed.ca nfs1
192.168.1.32 nfs2.lightspeed.ca nfs2
127.0.1.1 nfs3.lightspeed.ca nfs3
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
I can ping the hostname:
root@nfs3:/home/ernied# ping -c 3 nfs1
PING nfs1.lightspeed.ca (192.168.1.31) 56(84) bytes of data.
64 bytes from nfs1.lightspeed.ca (192.168.1.31): icmp_seq=1 ttl=64
time=0.148 ms
64 bytes from nfs1.lightspeed.ca (192.168.1.31): icmp_seq=2 ttl=64
time=0.126 ms
64 bytes from nfs1.lightspeed.ca (192.168.1.31): icmp_seq=3 ttl=64
time=0.133 ms
--- nfs1.lightspeed.ca ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.126/0.135/0.148/0.016 ms
I can get gluster to probe the hostname:
root@nfs3:/home/ernied# gluster peer probe nfs1
peer probe: success. Host nfs1 port 24007 already in peer list
But if I try to create the brick on the new node, it says that the
host can't be found? Um...
root@nfs3:/home/ernied# gluster volume create gv2 replica 3
nfs1.lightspeed.ca:/brick1/gv2/ nfs2.lightspeed.ca:/brick1/gv2/
nfs3.lightspeed.ca:/brick1/gv2
volume create: gv2: failed: Failed to find host nfs1.lightspeed.ca
Our logs from /var/log/glusterfs/etc-glusterfs-glusterd.vol.log:
[2016-04-06 18:19:18.107459] E [MSGID: 106452]
[glusterd-utils.c:5825:glusterd_new_brick_validate] 0-management:
Failed to find host nfs1.lightspeed.ca
[2016-04-06 18:19:18.107496] E [MSGID: 106536]
[glusterd-volume-ops.c:1364:glusterd_op_stage_create_volume]
0-management: Failed to find host nfs1.lightspeed.ca
[2016-04-06 18:19:18.107516] E [MSGID: 106301]
[glusterd-syncop.c:1281:gd_stage_op_phase] 0-management: Staging of
operation 'Volume Create' failed on localhost : Failed to find host
nfs1.lightspeed.ca
[2016-04-06 18:19:18.231864] E [MSGID: 106170]
[glusterd-handshake.c:1051:gd_validate_mgmt_hndsk_req] 0-management:
Request from peer 192.168.1.31:65530 has an entry in peerinfo, but
uuid does not match
We have introduced a new check to reject a peer if the request is
coming
from a node where the hostname matches but UUID is different. This can
happen if a node goes through a re-installation and its
/var/lib/glusterd/* content is wiped off. Look at [1] for more details.
[1] http://review.gluster.org/13519
Do confirm if that's the case.
I couldn't say if that's *exactly* the case, but it's pretty close. I
don't recall ever removing /var/lib/glusterd/* or any of its contents,
but the operating system isn't exactly the way it was when I first tried
to add this node to the cluster.
What should I do to *fix* the problem though, so I can add this node to
the cluster? This bug report doesn't appear to provide a solution. I've
tried removing the node from the cluster, and that failed too. Things
seem to be in a very screwey state right now.
[2016-04-06 18:19:18.231919] E [MSGID: 106170]
[glusterd-handshake.c:1060:gd_validate_mgmt_hndsk_req] 0-management:
Rejecting management handshake request from unknown peer
192.168.1.31:65530
That error about the entry in peerinfo doesn't match anything in
Google besides the source code for Gluster. My guess is that my
earlier unsuccessful attempts to add this node before v3.7.10 have
created a conflict that needs to be cleared.
More interesting, is what happens when I try to add the third server
to
the brick from the first gluster server:
root@nfs1:/home/ernied# gluster volume add-brick gv2 replica 3
nfs3:/brick1/gv2
volume add-brick: failed: One or more nodes do not support the
required
op-version. Cluster op-version must atleast be 30600.
Yet, when I view the operating version in
/var/lib/glusterd/glusterd.info:
root@nfs1:/home/ernied# cat /var/lib/glusterd/glusterd.info
UUID=1207917a-23bc-4bae-8238-cd691b7082c7
operating-version=30501
root@nfs2:/home/ernied# cat /var/lib/glusterd/glusterd.info
UUID=e394fcec-41da-482a-9b30-089f717c5c06
operating-version=30501
root@nfs3:/home/ernied# cat /var/lib/glusterd/glusterd.info
UUID=ae191e96-9cd6-4e2b-acae-18f2cc45e6ed
operating-version=30501
I see that the operating version is the same on all nodes!
Here cluster op-version is pretty old. You need to make sure that you
bump up the op-version by 'gluster volume set all cluster.op-version
30710'. add-brick code path has a check that your cluster op-version
has
to be at least 30600 if you are with gluster version >=3.6 which is the
case here.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
