Tomoaki,
Can you send the info file under /etc/glusterd/vols/<volname>/ from
machines were the peer probe
command was issued and the 'peer' who is being rejected?
thanks,
kp
On 08/17/2011 12:10 PM, Tomoaki Sato wrote:
> Mohit
>
> (2011/08/17 13:53), Mohit Anchlia wrote:
>> Not sure. It could be because the new node doesn't have the volume
>> configs. Can you try gluster volume sync to sync the configs and see
>> if that helps?
>
> - at foo-1-private -
> gluster> volume sync foo-3-private
> please delete all the volumes before full sync
> gluster>
> --
>
> - at foo-3-private -
> gluster> volume sync foo foo-1-private
> sync from localhost not allowed
> gluster>
> --
>
> Question is "How should I add extra peers to existing file systems ?".
> extra peers = new nodes.
>
> Could you tell me right instructions to gluster probe new nodes after
> the volume starting ?
>
>>
>> Also, not sure why you are getting "Unable to find hostname:
>> foo-3-private"
>>
>
> "Unable to find hostname: foo-3-private" was printed out on both OK
> and NG cases.
> "Cksums of volume foo differ. local cksum = 1403573944, remote cksum =
> -1413994823" was printed out on NG case only.
>
> 'Peer in Cluster', OK case:
> [2011-08-17 15:08:47.462646] I
> [glusterd-handler.c:602:glusterd_handle_cli_probe] 0-glusterd:
> Received CLI probe req foo-3-private 24007
> [2011-08-17 15:08:47.466194] I
> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
> find hostname: foo-3-private
> [2011-08-17 15:08:47.466224] I
> [glusterd-handler.c:3120:glusterd_probe_begin] 0-glusterd: Unable to
> find peerinfo for host: foo-3-private (24007)
> [2011-08-17 15:08:47.469365] W
> [rpc-transport.c:728:rpc_transport_load] 0-rpc-transport: missing
> 'option transport-type'. defaulting to "socket"
> [2011-08-17 15:08:47.473671] I
> [glusterd-handler.c:3102:glusterd_friend_add] 0-glusterd: connect
> returned 0
> [2011-08-17 15:08:47.474167] I
> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
> Program glusterd clnt mgmt, Num (1238433), Version (1)
> [2011-08-17 15:08:47.474214] I
> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
> Friend foo-3-private found.. state: 0
> [2011-08-17 15:08:47.483485] I
> [glusterd-rpc-ops.c:364:glusterd3_1_probe_cbk] 0-glusterd: Received
> probe resp from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
> foo-3-private
> [2011-08-17 15:08:47.483516] I
> [glusterd-handler.c:379:glusterd_friend_find] 0-glusterd: Unable to
> find peer by uuid
> [2011-08-17 15:08:47.483562] I
> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
> Friend foo-3-private found.. state: 0
> [2011-08-17 15:08:47.483764] I
> [glusterd-rpc-ops.c:409:glusterd3_1_probe_cbk] 0-glusterd: Received
> resp to probe req
> [2011-08-17 15:08:47.484040] I
> [glusterd-rpc-ops.c:454:glusterd3_1_friend_add_cbk] 0-glusterd:
> Received ACC from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
> foo-3-private, port: 0
> [2011-08-17 15:08:47.484088] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Probe Sent to Peer
> [2011-08-17 15:08:47.484153] I
> [glusterd-handler.c:3293:glusterd_xfer_cli_probe_resp] 0-glusterd:
> Responded to CLI, ret: 0
> [2011-08-17 15:08:47.484554] I
> [glusterd-handler.c:2882:glusterd_handle_probe_query] 0-glusterd:
> Received probe from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f
> [2011-08-17 15:08:47.484585] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Accepted peer request
> [2011-08-17 15:08:47.484647] I
> [glusterd-handler.c:2917:glusterd_handle_probe_query] 0-glusterd:
> Responded to 192.168.1.129, op_ret: 0, op_errno: 0, ret: 0
> [2011-08-17 15:08:47.485499] I
> [glusterd-handler.c:2614:glusterd_handle_incoming_friend_req]
> 0-glusterd: Received probe from uuid:
> 464b3ea0-1b2b-4683-8209-72220dcb295f
> [2011-08-17 15:08:47.485536] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Accepted peer request
> [2011-08-17 15:08:47.485590] I
> [glusterd-handler.c:3270:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to 192.168.1.129 (0), ret: 0
> [2011-08-17 15:08:47.485713] I
> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
> 464b3ea0-1b2b-4683-8209-72220dcb295f, host: foo-3-private
> [2011-08-17 15:08:47.486203] I
> [glusterd-rpc-ops.c:636:glusterd3_1_friend_update_cbk] 0-glusterd:
> Received ACC from uuid:
> [2011-08-17 15:08:47.486259] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Peer in Cluster
> [2011-08-17 15:08:47.486284] I
> [glusterd-handler.c:2761:glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f
> [2011-08-17 15:08:47.486316] I
> [glusterd-handler.c:2806:glusterd_handle_friend_update] 0-: Received
> uuid: 4b5b0ecb-7d18-4ec4-90d9-0df2d392b63f, hostname:192.168.1.129
> [2011-08-17 15:08:47.486335] I
> [glusterd-handler.c:2809:glusterd_handle_friend_update] 0-: Received
> my uuid as Friend
>
>
> 'Peer Rejected', NG case:
> [2011-08-17 15:10:19.21262] I
> [glusterd-handler.c:602:glusterd_handle_cli_probe] 0-glusterd:
> Received CLI probe req foo-3-private 24007
> [2011-08-17 15:10:19.24605] I
> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
> find hostname: foo-3-private
> [2011-08-17 15:10:19.24648] I
> [glusterd-handler.c:3120:glusterd_probe_begin] 0-glusterd: Unable to
> find peerinfo for host: foo-3-private (24007)
> [2011-08-17 15:10:19.27736] W [rpc-transport.c:728:rpc_transport_load]
> 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
> [2011-08-17 15:10:19.32034] I
> [glusterd-handler.c:3102:glusterd_friend_add] 0-glusterd: connect
> returned 0
> [2011-08-17 15:10:19.32389] I
> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
> Program glusterd clnt mgmt, Num (1238433), Version (1)
> [2011-08-17 15:10:19.32426] I
> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
> Friend foo-3-private found.. state: 0
> [2011-08-17 15:10:19.40671] I
> [glusterd-rpc-ops.c:364:glusterd3_1_probe_cbk] 0-glusterd: Received
> probe resp from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
> foo-3-private
> [2011-08-17 15:10:19.40720] I
> [glusterd-handler.c:379:glusterd_friend_find] 0-glusterd: Unable to
> find peer by uuid
> [2011-08-17 15:10:19.40748] I
> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
> Friend foo-3-private found.. state: 0
> [2011-08-17 15:10:19.40983] I
> [glusterd-rpc-ops.c:409:glusterd3_1_probe_cbk] 0-glusterd: Received
> resp to probe req
> [2011-08-17 15:10:19.42854] I [rpc-clnt.c:696:rpc_clnt_handle_cbk]
> 0-rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program:
> 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
> [2011-08-17 15:10:19.50762] I
> [glusterd-rpc-ops.c:454:glusterd3_1_friend_add_cbk] 0-glusterd:
> Received ACC from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
> foo-3-private, port: 0
> [2011-08-17 15:10:19.50794] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Probe Sent to Peer
> [2011-08-17 15:10:19.50851] I
> [glusterd-handler.c:3293:glusterd_xfer_cli_probe_resp] 0-glusterd:
> Responded to CLI, ret: 0
> [2011-08-17 15:10:19.51413] I
> [glusterd-handler.c:2882:glusterd_handle_probe_query] 0-glusterd:
> Received probe from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f
> [2011-08-17 15:10:19.51444] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Accepted peer request
> [2011-08-17 15:10:19.51487] I
> [glusterd-handler.c:2917:glusterd_handle_probe_query] 0-glusterd:
> Responded to 192.168.1.129, op_ret: 0, op_errno: 0, ret: 0
> [2011-08-17 15:10:19.51853] I
> [glusterd-handler.c:2614:glusterd_handle_incoming_friend_req]
> 0-glusterd: Received probe from uuid:
> 464b3ea0-1b2b-4683-8209-72220dcb295f
> [2011-08-17 15:10:19.51885] I
> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
> Friend found... state: Accepted peer request
> [2011-08-17 15:10:19.51930] E
> [glusterd-utils.c:1407:glusterd_compare_friend_volume] 0-: Cksums of
> volume foo differ. local cksum = 1403573944, remote cksum = -1413994823
> [2011-08-17 15:10:19.51975] I
> [glusterd-handler.c:3270:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to 192.168.1.129 (0), ret: 0
>
>
>> On Tue, Aug 16, 2011 at 8:18 PM, Tomoaki Sato<tsato at valinux.co.jp>
>> wrote:
>>> Mohit,
>>>
>>> let me say again.
>>> 3.1.6-1 fail to 'peer probe' after 'start volume' in my environment.
>>> case-A) peer probe foo-3-private --> Peer in Cluster
>>>
>>> <delete all configuration files and reboot all foo-X-private nodes>
>>>
>>> [root at foo-1-private ~]# gluster peer probe foo-3-private
>>> Probe successful
>>> [root at foo-1-private ~]# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: foo-3-private
>>> Uuid: ef7d3c43-219a-4d13-a918-2639455cfbe7
>>> State: Peer in Cluster (Connected)
>>>
>>> case-B) create volume then peer probe foo-3-private --> Peer in
>>> Cluster
>>>
>>> <delete all configuration files and reboot all foo-X-private nodes>
>>>
>>> [root at foo-1-private ~]# gluster volume create foo
>>> foo-1-private:/mnt/brick
>>> Creation of volume foo has been successful. Please start the volume to
>>> access data.
>>> [root at foo-1-private ~]# gluster peer probe foo-3-private
>>> Probe successful
>>> [root at foo-1-private ~]# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: foo-3-private
>>> Uuid: fe44c954-4679-4389-a0e6-4c1fd4569a02
>>> State: Peer in Cluster (Connected)
>>>
>>> case-C) start volume then peer probe foo-3-private --> Peer Rejected
>>>
>>> <delete all configuration files and reboot all foo-X-private nodes>
>>>
>>> [root at foo-1-private ~]# gluster volume create foo
>>> foo-1-private:/mnt/brick
>>> Creation of volume foo has been successful. Please start the volume to
>>> access data.
>>> [root at foo-1-private ~]# gluster volume start foo
>>> Starting volume foo has been successful
>>> [root at foo-1-private ~]# gluster peer probe foo-3-private
>>> Probe successful
>>> [root at foo-1-private ~]# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: foo-3-private
>>> Uuid: bb6932e4-5bf0-4d34-872f-4a5fc1d0b6f8
>>> State: Peer Rejected (Connected)
>>>
>>>
>>>> Can you for now put it in /etc/hosts and test?
>>>
>>> All foo-X-private hosts have no entries in /etc/hosts.
>>> All the nodes obtain IP addresses from a DHCP server and register
>>> the IP
>>> address - host name paires to a DNS server.
>>>
>>>>
>>>> also, make sure you have same version of gluster running on all the
>>>> nodes.
>>>
>>> Since all three foo-X-private hosts are generated from a common VM
>>> template,
>>> same version of gluster run on all the nodes.
>>>
>>>>
>>>> What's the result of gluster peer status on node 3?
>>>
>>> [root at foo-1-private ~]# ssh foo-3-private gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: 192.168.1.129
>>> Uuid: 828bcc00-14d3-4505-8b35-d0ac6ca0730a
>>> State: Peer Rejected (Connected)
>>> [root at foo-1-private ~]#
>>>
>>>
>>> Best,
>>>
>>>
>>> (2011/08/17 0:53), Mohit Anchlia wrote:
>>>>
>>>> I see this in the logs:
>>>>
>>>> [2011-08-16 11:57:05.642903] I
>>>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
>>>> find hostname: foo-3-private
>>>>
>>>> Can you for now put it in /etc/hosts and test?
>>>>
>>>> also, make sure you have same version of gluster running on all the
>>>> nodes.
>>>>
>>>> What's the result of gluster peer status on node 3?
>>>>
>>>> On Mon, Aug 15, 2011 at 8:18 PM, Tomoaki
>>>> Sato<tsato at valinux.co.jp> wrote:
>>>>>
>>>>> Mohit
>>>>>
>>>>> I've tried same test and reproduce the 'Peer Rejected' status.
>>>>> please find config files and log files in attached taz.
>>>>>
>>>>>
>>>>> [root at vhead-010 ~]# date
>>>>> Tue Aug 16 11:55:15 JST 2011
>>>>> [root at vhead-010 ~]# cat a.sh
>>>>> #!/bin/bash
>>>>> for i in foo-{1..3}-private
>>>>> do
>>>>> ssh ${i} service glusterd stop
>>>>> ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>>>>> ssh ${i} rm -rf /etc/glusterd/vols/*
>>>>> ssh ${i} service iptables stop
>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/nfs.log
>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/bricks/mnt-brick.log
>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/.cmd_log_history
>>>>> ssh ${i} cp /dev/null
>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>> ssh ${i} service glusterd start
>>>>> ssh ${i} find /etc/glusterd
>>>>> ssh ${i} service glusterd status
>>>>> done
>>>>> [root at vhead-010 ~]# bash a.sh
>>>>> Stopping glusterd:[ OK ]
>>>>> Flushing firewall rules: [ OK ]
>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>> Unloading iptables modules: [ OK ]
>>>>> Starting glusterd:[ OK ]
>>>>> /etc/glusterd
>>>>> /etc/glusterd/glusterd.info
>>>>> /etc/glusterd/nfs
>>>>> /etc/glusterd/nfs/run
>>>>> /etc/glusterd/peers
>>>>> /etc/glusterd/vols
>>>>> glusterd (pid 15617) is running...
>>>>> Stopping glusterd:[ OK ]
>>>>> Flushing firewall rules: [ OK ]
>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>> Unloading iptables modules: [ OK ]
>>>>> Starting glusterd:[ OK ]
>>>>> /etc/glusterd
>>>>> /etc/glusterd/glusterd.info
>>>>> /etc/glusterd/nfs
>>>>> /etc/glusterd/nfs/run
>>>>> /etc/glusterd/peers
>>>>> /etc/glusterd/vols
>>>>> glusterd (pid 15147) is running...
>>>>> Stopping glusterd:[ OK ]
>>>>> Flushing firewall rules: [ OK ]
>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>> Unloading iptables modules: [ OK ]
>>>>> Starting glusterd:[ OK ]
>>>>> /etc/glusterd
>>>>> /etc/glusterd/glusterd.info
>>>>> /etc/glusterd/nfs
>>>>> /etc/glusterd/nfs/run
>>>>> /etc/glusterd/peers
>>>>> /etc/glusterd/vols
>>>>> glusterd (pid 15177) is running...
>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>> Last login: Tue Aug 16 09:51:49 2011 from dlp.local.valinux.co.jp
>>>>> [root at localhost ~]# gluster peer probe foo-2-private
>>>>> Probe successful
>>>>> [root at localhost ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: foo-2-private
>>>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>>> State: Peer in Cluster (Connected)
>>>>> [root at localhost ~]# gluster volume create foo
>>>>> foo-1-private:/mnt/brick
>>>>> Creation of volume foo has been successful. Please start the
>>>>> volume to
>>>>> access data.
>>>>> [root at localhost ~]# gluster volume start foo
>>>>> Starting volume foo has been successful
>>>>> [root at localhost ~]# gluster volume add-brick foo
>>>>> foo-2-private:/mnt/brick
>>>>> Add Brick successful
>>>>> [root at localhost ~]# gluster peer probe foo-3-private
>>>>> Probe successful
>>>>> [root at localhost ~]# gluster peer status
>>>>> Number of Peers: 2
>>>>>
>>>>> Hostname: foo-2-private
>>>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: foo-3-private
>>>>> Uuid: 7587ae34-9209-484a-9576-3939e061720c
>>>>> State: Peer Rejected (Connected)
>>>>> [root at localhost ~]# exit
>>>>> logout
>>>>> Connection to foo-1-private closed.
>>>>> [root at vhead-010 ~]# find foo_log_and_conf
>>>>> foo_log_and_conf
>>>>> foo_log_and_conf/foo-2-private
>>>>> foo_log_and_conf/foo-2-private/glusterd
>>>>> foo_log_and_conf/foo-2-private/glusterd/glusterd.info
>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs
>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/nfs-server.vol
>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run
>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run/nfs.pid
>>>>> foo_log_and_conf/foo-2-private/glusterd/peers
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/info
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/cksum
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run/foo-2-private-mnt-brick.pid
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo-fuse.vol
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>>>>
>>>>> foo_log_and_conf/foo-2-private/glusterfs
>>>>> foo_log_and_conf/foo-2-private/glusterfs/nfs.log
>>>>> foo_log_and_conf/foo-2-private/glusterfs/bricks
>>>>> foo_log_and_conf/foo-2-private/glusterfs/bricks/mnt-brick.log
>>>>> foo_log_and_conf/foo-2-private/glusterfs/.cmd_log_history
>>>>> foo_log_and_conf/foo-2-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>
>>>>> foo_log_and_conf/foo-1-private
>>>>> foo_log_and_conf/foo-1-private/glusterd
>>>>> foo_log_and_conf/foo-1-private/glusterd/glusterd.info
>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs
>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/nfs-server.vol
>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run
>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run/nfs.pid
>>>>> foo_log_and_conf/foo-1-private/glusterd/peers
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/peers/20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/peers/7587ae34-9209-484a-9576-3939e061720c
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/info
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/cksum
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run/foo-1-private-mnt-brick.pid
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo-fuse.vol
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>>>>
>>>>> foo_log_and_conf/foo-1-private/glusterfs
>>>>> foo_log_and_conf/foo-1-private/glusterfs/nfs.log
>>>>> foo_log_and_conf/foo-1-private/glusterfs/bricks
>>>>> foo_log_and_conf/foo-1-private/glusterfs/bricks/mnt-brick.log
>>>>> foo_log_and_conf/foo-1-private/glusterfs/.cmd_log_history
>>>>> foo_log_and_conf/foo-1-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>
>>>>> foo_log_and_conf/foo-3-private
>>>>> foo_log_and_conf/foo-3-private/glusterd
>>>>> foo_log_and_conf/foo-3-private/glusterd/glusterd.info
>>>>> foo_log_and_conf/foo-3-private/glusterd/nfs
>>>>> foo_log_and_conf/foo-3-private/glusterd/nfs/run
>>>>> foo_log_and_conf/foo-3-private/glusterd/peers
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/info
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>>>
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/cksum
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo-fuse.vol
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>>>>
>>>>> foo_log_and_conf/foo-3-private/glusterfs
>>>>> foo_log_and_conf/foo-3-private/glusterfs/nfs.log
>>>>> foo_log_and_conf/foo-3-private/glusterfs/bricks
>>>>> foo_log_and_conf/foo-3-private/glusterfs/bricks/mnt-brick.log
>>>>> foo_log_and_conf/foo-3-private/glusterfs/.cmd_log_history
>>>>> foo_log_and_conf/foo-3-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>
>>>>> [root at vhead-010 ~]# exit
>>>>>
>>>>> Best,
>>>>>
>>>>> (2011/08/16 9:35), Mohit Anchlia wrote:
>>>>>>
>>>>>> I should have also asked you to stop and delete volume before
>>>>>> getting
>>>>>> rid of gluster config files. Can you get rid of directories also
>>>>>> inside vols and try to restart? It's trying to look for volume files
>>>>>> that we just removed.
>>>>>>
>>>>>> Also, just disable iptables for now explicitly.
>>>>>>
>>>>>> On Mon, Aug 15, 2011 at 5:22 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>>> wrote:
>>>>>>>
>>>>>>>> 1) run peer detach for all the servers
>>>>>>>
>>>>>>> done.
>>>>>>>
>>>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and
>>>>>>>> iptables
>>>>>>>> are turned off.
>>>>>>>
>>>>>>> done.
>>>>>>> by the way, the same test on 3.1.5-1 works fine with same
>>>>>>> environment.
>>>>>>>
>>>>>>>> 3) remove config files under /etc/glusterd
>>>>>>>
>>>>>>> please review following logs.
>>>>>>>
>>>>>>>> 4) run your tests again.
>>>>>>>
>>>>>>> I don't know why but glusterd service failed to start on all 3
>>>>>>> hosts.
>>>>>>>
>>>>>>> [root at vhead-010 ~]# date
>>>>>>> Tue Aug 16 09:12:53 JST 2011
>>>>>>> [root at vhead-010 ~]# cat a.sh
>>>>>>> #!/bin/bash
>>>>>>> for i in foo-{1..3}-private
>>>>>>> do
>>>>>>> ssh ${i} service glusterd stop
>>>>>>> ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>>>>>>> ssh ${i} service iptables restart
>>>>>>> ssh ${i} iptables -vL
>>>>>>> ssh ${i} service glusterd start
>>>>>>> ssh ${i} find /etc/glusterd
>>>>>>> ssh ${i} service glusterd status
>>>>>>> done
>>>>>>> [root at vhead-010 ~]# bash a.sh
>>>>>>> Stopping glusterd:[ OK ]
>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>>
>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>>
>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>> Starting glusterd:[ OK ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> /etc/glusterd/vols/foo
>>>>>>> /etc/glusterd/vols/foo/bricks
>>>>>>> /etc/glusterd/vols/foo/run
>>>>>>> glusterd is stopped
>>>>>>> Stopping glusterd:[ OK ]
>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>>
>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>>
>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>> Starting glusterd:[ OK ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> /etc/glusterd/vols/foo
>>>>>>> /etc/glusterd/vols/foo/bricks
>>>>>>> /etc/glusterd/vols/foo/run
>>>>>>> glusterd is stopped
>>>>>>> Stopping glusterd:[ OK ]
>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>>
>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>>
>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>> pkts bytes target prot opt in out source
>>>>>>> destination
>>>>>>> Starting glusterd:[ OK ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> /etc/glusterd/vols/foo
>>>>>>> /etc/glusterd/vols/foo/bricks
>>>>>>> /etc/glusterd/vols/foo/run
>>>>>>> glusterd is stopped
>>>>>>> [root at vhead-010 ~]# date
>>>>>>> Tue Aug 16 09:13:20 JST 2011
>>>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>>>> Last login: Tue Aug 16 09:06:57 2011 from dlp.local.valinux.co.jp
>>>>>>> [root at localhost ~]# tail -20
>>>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>>> ...
>>>>>>> [2011-08-16 09:13:01.85858] I [glusterd.c:304:init]
>>>>>>> 0-management: Using
>>>>>>> /etc/glusterd as working directory
>>>>>>> [2011-08-16 09:13:01.87294] E
>>>>>>> [rpc-transport.c:799:rpc_transport_load]
>>>>>>> 0-rpc-transport:
>>>>>>> /opt/glusterfs/3.1.6/lib64/glusterfs/3.1.6/rpc-transport/rdma.so:
>>>>>>> cannot
>>>>>>> open shared object file: No such file or directory
>>>>>>> [2011-08-16 09:13:01.87340] E
>>>>>>> [rpc-transport.c:803:rpc_transport_load]
>>>>>>> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma'
>>>>>>> is not
>>>>>>> valid or not found on this machine
>>>>>>> [2011-08-16 09:13:01.87402] E
>>>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd:
>>>>>>> Unable
>>>>>>> to
>>>>>>> retrieve store handle for /etc/glusterd/glusterd.info, error: No
>>>>>>> such
>>>>>>> file
>>>>>>> or directory
>>>>>>> [2011-08-16 09:13:01.87422] E
>>>>>>> [glusterd-store.c:761:glusterd_retrieve_uuid]
>>>>>>> 0-: Unable to get store handle!
>>>>>>> [2011-08-16 09:13:01.87514] I [glusterd.c:95:glusterd_uuid_init]
>>>>>>> 0-glusterd:
>>>>>>> generated UUID: c0cef9f9-a79e-4189-8955-d83927db9cee
>>>>>>> [2011-08-16 09:13:01.87681] E
>>>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd:
>>>>>>> Unable
>>>>>>> to
>>>>>>> retrieve store handle for /etc/glusterd/vols/foo/info, error: No
>>>>>>> such
>>>>>>> file
>>>>>>> or directory
>>>>>>> [2011-08-16 09:13:01.87704] E
>>>>>>> [glusterd-store.c:1328:glusterd_store_retrieve_volumes] 0-:
>>>>>>> Unable to
>>>>>>> restore volume: foo
>>>>>>
>>>>>>> [2011-08-16 09:13:01.87732] E [xlator.c:843:xlator_init]
>>>>>>> 0-management:
>>>>>>> Initialization of volume 'management' failed, review your
>>>>>>> volfile again
>>>>>>> [2011-08-16 09:13:01.87751] E [graph.c:331:glusterfs_graph_init]
>>>>>>> 0-management: initializing translator failed
>>>>>>> [2011-08-16 09:13:01.87818] I [glusterfsd.c:712:cleanup_and_exit]
>>>>>>> 0-glusterfsd: shutting down
>>>>>>> [root at localhost ~]# exit
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> (2011/08/16 8:52), Mohit Anchlia wrote:
>>>>>>>>
>>>>>>>> Logs are generally in /var/log/gluster
>>>>>>>>
>>>>>>>> Since you are playing with it. I would suggest this:
>>>>>>>>
>>>>>>>> 1) run peer detach for all the servers
>>>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and
>>>>>>>> iptables
>>>>>>>> are turned off.
>>>>>>>> 3) remove config files under /etc/glusterd
>>>>>>>> 4) run your tests again.
>>>>>>>>
>>>>>>>> On Mon, Aug 15, 2011 at 4:28 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Thanks, Mohit
>>>>>>>>>
>>>>>>>>> (2011/08/16 8:05), Mohit Anchlia wrote:
>>>>>>>>>>
>>>>>>>>>> What's in your logs?
>>>>>>>>>
>>>>>>>>> I can obtain logs needed. could you tell me the instruction to
>>>>>>>>> take
>>>>>>>>> the
>>>>>>>>> logs?
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Did you have foo-3-private before in your gluster cluster
>>>>>>>>>> ever or
>>>>>>>>>> adding this host for the first time?
>>>>>>>>>
>>>>>>>>> It was first time.
>>>>>>>>> All foo-X-private has no entries in /etc/glusterd/peers/ and
>>>>>>>>> /etc/glusterd/vols/.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Try gluster peer detach and then remove any left over
>>>>>>>>>> configuration
>>>>>>>>>> in
>>>>>>>>>> /etc/glusterd config directory. After that try again and see
>>>>>>>>>> if that
>>>>>>>>>> works.
>>>>>>>>>
>>>>>>>>> [root at vhead-010 ~]# date
>>>>>>>>> Tue Aug 16 08:17:49 JST 2011
>>>>>>>>> [root at vhead-010 ~]# cat a.sh
>>>>>>>>> #!/bin/bash
>>>>>>>>> for i in foo-{1..3}-private
>>>>>>>>> do
>>>>>>>>> ssh ${i} service glusterd stop
>>>>>>>>> ssh ${i} rm -rf /etc/glusterd/peers/*
>>>>>>>>> ssh ${i} rm -rf /etc/glusterd/vols/*
>>>>>>>>> ssh ${i} service glusterd start
>>>>>>>>> ssh ${i} find /etc/glusterd
>>>>>>>>> done
>>>>>>>>> [root at vhead-010 ~]# bash a.sh
>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>> /etc/glusterd
>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>> /etc/glusterd/peers
>>>>>>>>> /etc/glusterd/vols
>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>> /etc/glusterd
>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>> /etc/glusterd/peers
>>>>>>>>> /etc/glusterd/vols
>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>> /etc/glusterd
>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>> /etc/glusterd/peers
>>>>>>>>> /etc/glusterd/vols
>>>>>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>>>>>> [root at localhost ~]# gluster peer probe foo-2-private
>>>>>>>>> Probe successful
>>>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>>>> Number of Peers: 1
>>>>>>>>>
>>>>>>>>> Hostname: foo-2-private
>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>> [root at localhost ~]# gluster volume create foo
>>>>>>>>> foo-1-private:/mnt/brick
>>>>>>>>> Creation of volume foo has been successful. Please start the
>>>>>>>>> volume
>>>>>>>>> to
>>>>>>>>> access da
>>>>>>>>> ta.
>>>>>>>>> [root at localhost ~]# gluster volume start foo
>>>>>>>>> Starting volume foo has been successful
>>>>>>>>> [root at localhost ~]# gluster volume add-brick foo
>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>> Add Brick successful
>>>>>>>>> [root at localhost ~]# gluster peer probe foo-3-private
>>>>>>>>> Probe successful
>>>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>>>> Number of Peers: 2
>>>>>>>>>
>>>>>>>>> Hostname: foo-2-private
>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>
>>>>>>>>> Hostname: foo-3-private
>>>>>>>>> Uuid: 7fb98dac-fef7-4b33-837c-6483a767ec3e
>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>> [root at localhost ~]# cat /var/log/glusterfs/.cmd_log_history
>>>>>>>>> ...
>>>>>>>>> [2011-08-16 08:20:28.862619] peer probe : on host
>>>>>>>>> foo-2-private:24007
>>>>>>>>> [2011-08-16 08:20:28.912419] peer probe : on host
>>>>>>>>> foo-2-private:24007
>>>>>>>>> FAILED
>>>>>>>>> [2011-08-16 08:20:58.382350] Volume create : on volname: foo
>>>>>>>>> attempted
>>>>>>>>> [2011-08-16 08:20:58.382461] Volume create : on volname: foo
>>>>>>>>> type:DEFAULT
>>>>>>>>> count:
>>>>>>>>> 1 bricks: foo-1-private:/mnt/brick
>>>>>>>>> [2011-08-16 08:20:58.384674] Volume create : on volname: foo
>>>>>>>>> SUCCESS
>>>>>>>>> [2011-08-16 08:21:04.831772] volume start : on volname: foo
>>>>>>>>> SUCCESS
>>>>>>>>> [2011-08-16 08:21:22.682292] Volume add-brick : on volname: foo
>>>>>>>>> attempted
>>>>>>>>> [2011-08-16 08:21:22.682385] Volume add-brick : volname: foo type
>>>>>>>>> DEFAULT
>>>>>>>>> count:
>>>>>>>>> 1 bricks: foo-2-private:/mnt/brick
>>>>>>>>> [2011-08-16 08:21:22.682499] Volume add-brick : on volname: foo
>>>>>>>>> SUCCESS
>>>>>>>>> [2011-08-16 08:21:39.124574] peer probe : on host
>>>>>>>>> foo-3-private:24007
>>>>>>>>> [2011-08-16 08:21:39.135609] peer probe : on host
>>>>>>>>> foo-3-private:24007
>>>>>>>>> FAILED
>>>>>>>>>
>>>>>>>>> Tomo
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 15, 2011 at 3:37 PM, Tomoaki
>>>>>>>>>> Sato<tsato at valinux.co.jp>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> following instructions work fine with 3.1.5-1 but with 3.1.6-1.
>>>>>>>>>>>
>>>>>>>>>>> 1. make a new file system without peers. [OK]
>>>>>>>>>>>
>>>>>>>>>>> foo-1-private# gluster volume create foo
>>>>>>>>>>> foo-1-private:/mnt/brick
>>>>>>>>>>> foo-1-private# gluster volume start foo
>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>> No peers present
>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>
>>>>>>>>>>> 2. add a peer to the file system. [NG]
>>>>>>>>>>>
>>>>>>>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>>>>>>>> Probe successful
>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>> Number of Peers: 1
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>>>> Host foo-2-private not connected
>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> following instructions work fine even with 3.1.6-1.
>>>>>>>>>>>
>>>>>>>>>>> 1. make a new file system with single peer. [OK]
>>>>>>>>>>>
>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>> No peer presents
>>>>>>>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>>>>>>>> Probe successful
>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>> Number of Peers: 1
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>> foo-1-private# gluster volume create foo
>>>>>>>>>>> foo-1-private:/mnt/brick
>>>>>>>>>>> Creation of volume foo has been successful. Please start the
>>>>>>>>>>> volume
>>>>>>>>>>> to
>>>>>>>>>>> access data.
>>>>>>>>>>> foo-1-private# gluster volume start foo
>>>>>>>>>>> Starting volume foo has been successful
>>>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>>>> Add Brick successful
>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>
>>>>>>>>>>> But ...
>>>>>>>>>>>
>>>>>>>>>>> 2. add a peer to the file system. [NG]
>>>>>>>>>>>
>>>>>>>>>>> foo-1-private# gluster peer probe foo-3-private
>>>>>>>>>>> Probe successful
>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>> Number of Peers: 2
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-3-private
>>>>>>>>>>> Uuid: 7fb98dac-fef704b33-837c-6483a767ec3e
>>>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>>>> foo-3-private:/mnt/brick
>>>>>>>>>>> Host foo-3-private not connected
>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>
>>>>>>>>>>> How should I add extra peers to existing file systems ?
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
