Tomoaki, You seem to have hit a bug in peer probe operation. This will be tracked in bug 3432: see http://bugs.gluster.com/show_bug.cgi?id=3432 thanks, kp On 08/17/2011 12:28 PM, Tomoaki Sato wrote: > kp, > > please find attached tar files. > > [root at vhead-010 ~]# ssh foo-1-private > [root at foo-1-private ~]# gluster volume create foo > foo-1-private:/mnt/brick > Creation of volume foo has been successful. Please start the volume to > access data. > [root at foo-1-private ~]# gluster volume start foo > Starting volume foo has been successful > [root at foo-1-private ~]# gluster peer probe foo-3-private > Probe successful > [root at foo-1-private ~]# gluster peer status > Number of Peers: 1 > > Hostname: foo-3-private > Uuid: 0c76c6ec-a7ea-405e-9d1d-8e100e155ec3 > State: Peer Rejected (Connected) > [root at foo-1-private ~]# cd /etc/glusterd/vols/ > [root at foo-1-private vols]# tar cf /tmp/foo-1-private.tar foo > [root at foo-1-private vols]# ssh foo-3-private > [root at foo-3-private ~]# cd /etc/glusterd/vols > [root at foo-3-private vols]# tar cf /tmp/foo-3-private.tar foo > [root at foo-3-private vols]# > > Thanks, > Tomo > > (2011/08/17 15:46), krish wrote: >> Tomoaki, >> >> Can you send the info file under /etc/glusterd/vols/<volname>/ from >> machines were the peer probe >> command was issued and the 'peer' who is being rejected? >> >> thanks, >> kp >> >> On 08/17/2011 12:10 PM, Tomoaki Sato wrote: >>> Mohit >>> >>> (2011/08/17 13:53), Mohit Anchlia wrote: >>>> Not sure. It could be because the new node doesn't have the volume >>>> configs. Can you try gluster volume sync to sync the configs and see >>>> if that helps? >>> >>> - at foo-1-private - >>> gluster> volume sync foo-3-private >>> please delete all the volumes before full sync >>> gluster> >>> -- >>> >>> - at foo-3-private - >>> gluster> volume sync foo foo-1-private >>> sync from localhost not allowed >>> gluster> >>> -- >>> >>> Question is "How should I add extra peers to existing file systems ?". >>> extra peers = new nodes. >>> >>> Could you tell me right instructions to gluster probe new nodes >>> after the volume starting ? >>> >>>> >>>> Also, not sure why you are getting "Unable to find hostname: >>>> foo-3-private" >>>> >>> >>> "Unable to find hostname: foo-3-private" was printed out on both OK >>> and NG cases. >>> "Cksums of volume foo differ. local cksum = 1403573944, remote cksum >>> = -1413994823" was printed out on NG case only. >>> >>> 'Peer in Cluster', OK case: >>> [2011-08-17 15:08:47.462646] I >>> [glusterd-handler.c:602:glusterd_handle_cli_probe] 0-glusterd: >>> Received CLI probe req foo-3-private 24007 >>> [2011-08-17 15:08:47.466194] I >>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to >>> find hostname: foo-3-private >>> [2011-08-17 15:08:47.466224] I >>> [glusterd-handler.c:3120:glusterd_probe_begin] 0-glusterd: Unable to >>> find peerinfo for host: foo-3-private (24007) >>> [2011-08-17 15:08:47.469365] W >>> [rpc-transport.c:728:rpc_transport_load] 0-rpc-transport: missing >>> 'option transport-type'. defaulting to "socket" >>> [2011-08-17 15:08:47.473671] I >>> [glusterd-handler.c:3102:glusterd_friend_add] 0-glusterd: connect >>> returned 0 >>> [2011-08-17 15:08:47.474167] I >>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using >>> Program glusterd clnt mgmt, Num (1238433), Version (1) >>> [2011-08-17 15:08:47.474214] I >>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd: >>> Friend foo-3-private found.. state: 0 >>> [2011-08-17 15:08:47.483485] I >>> [glusterd-rpc-ops.c:364:glusterd3_1_probe_cbk] 0-glusterd: Received >>> probe resp from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host: >>> foo-3-private >>> [2011-08-17 15:08:47.483516] I >>> [glusterd-handler.c:379:glusterd_friend_find] 0-glusterd: Unable to >>> find peer by uuid >>> [2011-08-17 15:08:47.483562] I >>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd: >>> Friend foo-3-private found.. state: 0 >>> [2011-08-17 15:08:47.483764] I >>> [glusterd-rpc-ops.c:409:glusterd3_1_probe_cbk] 0-glusterd: Received >>> resp to probe req >>> [2011-08-17 15:08:47.484040] I >>> [glusterd-rpc-ops.c:454:glusterd3_1_friend_add_cbk] 0-glusterd: >>> Received ACC from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host: >>> foo-3-private, port: 0 >>> [2011-08-17 15:08:47.484088] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Probe Sent to Peer >>> [2011-08-17 15:08:47.484153] I >>> [glusterd-handler.c:3293:glusterd_xfer_cli_probe_resp] 0-glusterd: >>> Responded to CLI, ret: 0 >>> [2011-08-17 15:08:47.484554] I >>> [glusterd-handler.c:2882:glusterd_handle_probe_query] 0-glusterd: >>> Received probe from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f >>> [2011-08-17 15:08:47.484585] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Accepted peer request >>> [2011-08-17 15:08:47.484647] I >>> [glusterd-handler.c:2917:glusterd_handle_probe_query] 0-glusterd: >>> Responded to 192.168.1.129, op_ret: 0, op_errno: 0, ret: 0 >>> [2011-08-17 15:08:47.485499] I >>> [glusterd-handler.c:2614:glusterd_handle_incoming_friend_req] >>> 0-glusterd: Received probe from uuid: >>> 464b3ea0-1b2b-4683-8209-72220dcb295f >>> [2011-08-17 15:08:47.485536] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Accepted peer request >>> [2011-08-17 15:08:47.485590] I >>> [glusterd-handler.c:3270:glusterd_xfer_friend_add_resp] 0-glusterd: >>> Responded to 192.168.1.129 (0), ret: 0 >>> [2011-08-17 15:08:47.485713] I >>> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: >>> 464b3ea0-1b2b-4683-8209-72220dcb295f, host: foo-3-private >>> [2011-08-17 15:08:47.486203] I >>> [glusterd-rpc-ops.c:636:glusterd3_1_friend_update_cbk] 0-glusterd: >>> Received ACC from uuid: >>> [2011-08-17 15:08:47.486259] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Peer in Cluster >>> [2011-08-17 15:08:47.486284] I >>> [glusterd-handler.c:2761:glusterd_handle_friend_update] 0-glusterd: >>> Received friend update from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f >>> [2011-08-17 15:08:47.486316] I >>> [glusterd-handler.c:2806:glusterd_handle_friend_update] 0-: Received >>> uuid: 4b5b0ecb-7d18-4ec4-90d9-0df2d392b63f, hostname:192.168.1.129 >>> [2011-08-17 15:08:47.486335] I >>> [glusterd-handler.c:2809:glusterd_handle_friend_update] 0-: Received >>> my uuid as Friend >>> >>> >>> 'Peer Rejected', NG case: >>> [2011-08-17 15:10:19.21262] I >>> [glusterd-handler.c:602:glusterd_handle_cli_probe] 0-glusterd: >>> Received CLI probe req foo-3-private 24007 >>> [2011-08-17 15:10:19.24605] I >>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to >>> find hostname: foo-3-private >>> [2011-08-17 15:10:19.24648] I >>> [glusterd-handler.c:3120:glusterd_probe_begin] 0-glusterd: Unable to >>> find peerinfo for host: foo-3-private (24007) >>> [2011-08-17 15:10:19.27736] W >>> [rpc-transport.c:728:rpc_transport_load] 0-rpc-transport: missing >>> 'option transport-type'. defaulting to "socket" >>> [2011-08-17 15:10:19.32034] I >>> [glusterd-handler.c:3102:glusterd_friend_add] 0-glusterd: connect >>> returned 0 >>> [2011-08-17 15:10:19.32389] I >>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using >>> Program glusterd clnt mgmt, Num (1238433), Version (1) >>> [2011-08-17 15:10:19.32426] I >>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd: >>> Friend foo-3-private found.. state: 0 >>> [2011-08-17 15:10:19.40671] I >>> [glusterd-rpc-ops.c:364:glusterd3_1_probe_cbk] 0-glusterd: Received >>> probe resp from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host: >>> foo-3-private >>> [2011-08-17 15:10:19.40720] I >>> [glusterd-handler.c:379:glusterd_friend_find] 0-glusterd: Unable to >>> find peer by uuid >>> [2011-08-17 15:10:19.40748] I >>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd: >>> Friend foo-3-private found.. state: 0 >>> [2011-08-17 15:10:19.40983] I >>> [glusterd-rpc-ops.c:409:glusterd3_1_probe_cbk] 0-glusterd: Received >>> resp to probe req >>> [2011-08-17 15:10:19.42854] I [rpc-clnt.c:696:rpc_clnt_handle_cbk] >>> 0-rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: >>> 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management) >>> [2011-08-17 15:10:19.50762] I >>> [glusterd-rpc-ops.c:454:glusterd3_1_friend_add_cbk] 0-glusterd: >>> Received ACC from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host: >>> foo-3-private, port: 0 >>> [2011-08-17 15:10:19.50794] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Probe Sent to Peer >>> [2011-08-17 15:10:19.50851] I >>> [glusterd-handler.c:3293:glusterd_xfer_cli_probe_resp] 0-glusterd: >>> Responded to CLI, ret: 0 >>> [2011-08-17 15:10:19.51413] I >>> [glusterd-handler.c:2882:glusterd_handle_probe_query] 0-glusterd: >>> Received probe from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f >>> [2011-08-17 15:10:19.51444] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Accepted peer request >>> [2011-08-17 15:10:19.51487] I >>> [glusterd-handler.c:2917:glusterd_handle_probe_query] 0-glusterd: >>> Responded to 192.168.1.129, op_ret: 0, op_errno: 0, ret: 0 >>> [2011-08-17 15:10:19.51853] I >>> [glusterd-handler.c:2614:glusterd_handle_incoming_friend_req] >>> 0-glusterd: Received probe from uuid: >>> 464b3ea0-1b2b-4683-8209-72220dcb295f >>> [2011-08-17 15:10:19.51885] I >>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd: >>> Friend found... state: Accepted peer request >>> [2011-08-17 15:10:19.51930] E >>> [glusterd-utils.c:1407:glusterd_compare_friend_volume] 0-: Cksums of >>> volume foo differ. local cksum = 1403573944, remote cksum = -1413994823 >>> [2011-08-17 15:10:19.51975] I >>> [glusterd-handler.c:3270:glusterd_xfer_friend_add_resp] 0-glusterd: >>> Responded to 192.168.1.129 (0), ret: 0 >>> >>> >>>> On Tue, Aug 16, 2011 at 8:18 PM, Tomoaki Sato<tsato at valinux.co.jp> >>>> wrote: >>>>> Mohit, >>>>> >>>>> let me say again. >>>>> 3.1.6-1 fail to 'peer probe' after 'start volume' in my environment. >>>>> case-A) peer probe foo-3-private --> Peer in Cluster >>>>> >>>>> <delete all configuration files and reboot all foo-X-private nodes> >>>>> >>>>> [root at foo-1-private ~]# gluster peer probe foo-3-private >>>>> Probe successful >>>>> [root at foo-1-private ~]# gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: foo-3-private >>>>> Uuid: ef7d3c43-219a-4d13-a918-2639455cfbe7 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> case-B) create volume then peer probe foo-3-private --> Peer in >>>>> Cluster >>>>> >>>>> <delete all configuration files and reboot all foo-X-private nodes> >>>>> >>>>> [root at foo-1-private ~]# gluster volume create foo >>>>> foo-1-private:/mnt/brick >>>>> Creation of volume foo has been successful. Please start the >>>>> volume to >>>>> access data. >>>>> [root at foo-1-private ~]# gluster peer probe foo-3-private >>>>> Probe successful >>>>> [root at foo-1-private ~]# gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: foo-3-private >>>>> Uuid: fe44c954-4679-4389-a0e6-4c1fd4569a02 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> case-C) start volume then peer probe foo-3-private --> Peer Rejected >>>>> >>>>> <delete all configuration files and reboot all foo-X-private nodes> >>>>> >>>>> [root at foo-1-private ~]# gluster volume create foo >>>>> foo-1-private:/mnt/brick >>>>> Creation of volume foo has been successful. Please start the >>>>> volume to >>>>> access data. >>>>> [root at foo-1-private ~]# gluster volume start foo >>>>> Starting volume foo has been successful >>>>> [root at foo-1-private ~]# gluster peer probe foo-3-private >>>>> Probe successful >>>>> [root at foo-1-private ~]# gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: foo-3-private >>>>> Uuid: bb6932e4-5bf0-4d34-872f-4a5fc1d0b6f8 >>>>> State: Peer Rejected (Connected) >>>>> >>>>> >>>>>> Can you for now put it in /etc/hosts and test? >>>>> >>>>> All foo-X-private hosts have no entries in /etc/hosts. >>>>> All the nodes obtain IP addresses from a DHCP server and register >>>>> the IP >>>>> address - host name paires to a DNS server. >>>>> >>>>>> >>>>>> also, make sure you have same version of gluster running on all >>>>>> the nodes. >>>>> >>>>> Since all three foo-X-private hosts are generated from a common VM >>>>> template, >>>>> same version of gluster run on all the nodes. >>>>> >>>>>> >>>>>> What's the result of gluster peer status on node 3? >>>>> >>>>> [root at foo-1-private ~]# ssh foo-3-private gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: 192.168.1.129 >>>>> Uuid: 828bcc00-14d3-4505-8b35-d0ac6ca0730a >>>>> State: Peer Rejected (Connected) >>>>> [root at foo-1-private ~]# >>>>> >>>>> >>>>> Best, >>>>> >>>>> >>>>> (2011/08/17 0:53), Mohit Anchlia wrote: >>>>>> >>>>>> I see this in the logs: >>>>>> >>>>>> [2011-08-16 11:57:05.642903] I >>>>>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to >>>>>> find hostname: foo-3-private >>>>>> >>>>>> Can you for now put it in /etc/hosts and test? >>>>>> >>>>>> also, make sure you have same version of gluster running on all >>>>>> the nodes. >>>>>> >>>>>> What's the result of gluster peer status on node 3? >>>>>> >>>>>> On Mon, Aug 15, 2011 at 8:18 PM, Tomoaki >>>>>> Sato<tsato at valinux.co.jp> wrote: >>>>>>> >>>>>>> Mohit >>>>>>> >>>>>>> I've tried same test and reproduce the 'Peer Rejected' status. >>>>>>> please find config files and log files in attached taz. >>>>>>> >>>>>>> >>>>>>> [root at vhead-010 ~]# date >>>>>>> Tue Aug 16 11:55:15 JST 2011 >>>>>>> [root at vhead-010 ~]# cat a.sh >>>>>>> #!/bin/bash >>>>>>> for i in foo-{1..3}-private >>>>>>> do >>>>>>> ssh ${i} service glusterd stop >>>>>>> ssh ${i} 'find /etc/glusterd -type f|xargs rm -f' >>>>>>> ssh ${i} rm -rf /etc/glusterd/vols/* >>>>>>> ssh ${i} service iptables stop >>>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/nfs.log >>>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/bricks/mnt-brick.log >>>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/.cmd_log_history >>>>>>> ssh ${i} cp /dev/null >>>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log >>>>>>> ssh ${i} service glusterd start >>>>>>> ssh ${i} find /etc/glusterd >>>>>>> ssh ${i} service glusterd status >>>>>>> done >>>>>>> [root at vhead-010 ~]# bash a.sh >>>>>>> Stopping glusterd:[ OK ] >>>>>>> Flushing firewall rules: [ OK ] >>>>>>> Setting chains to policy ACCEPT: filter [ OK ] >>>>>>> Unloading iptables modules: [ OK ] >>>>>>> Starting glusterd:[ OK ] >>>>>>> /etc/glusterd >>>>>>> /etc/glusterd/glusterd.info >>>>>>> /etc/glusterd/nfs >>>>>>> /etc/glusterd/nfs/run >>>>>>> /etc/glusterd/peers >>>>>>> /etc/glusterd/vols >>>>>>> glusterd (pid 15617) is running... >>>>>>> Stopping glusterd:[ OK ] >>>>>>> Flushing firewall rules: [ OK ] >>>>>>> Setting chains to policy ACCEPT: filter [ OK ] >>>>>>> Unloading iptables modules: [ OK ] >>>>>>> Starting glusterd:[ OK ] >>>>>>> /etc/glusterd >>>>>>> /etc/glusterd/glusterd.info >>>>>>> /etc/glusterd/nfs >>>>>>> /etc/glusterd/nfs/run >>>>>>> /etc/glusterd/peers >>>>>>> /etc/glusterd/vols >>>>>>> glusterd (pid 15147) is running... >>>>>>> Stopping glusterd:[ OK ] >>>>>>> Flushing firewall rules: [ OK ] >>>>>>> Setting chains to policy ACCEPT: filter [ OK ] >>>>>>> Unloading iptables modules: [ OK ] >>>>>>> Starting glusterd:[ OK ] >>>>>>> /etc/glusterd >>>>>>> /etc/glusterd/glusterd.info >>>>>>> /etc/glusterd/nfs >>>>>>> /etc/glusterd/nfs/run >>>>>>> /etc/glusterd/peers >>>>>>> /etc/glusterd/vols >>>>>>> glusterd (pid 15177) is running... >>>>>>> [root at vhead-010 ~]# ssh foo-1-private >>>>>>> Last login: Tue Aug 16 09:51:49 2011 from dlp.local.valinux.co.jp >>>>>>> [root at localhost ~]# gluster peer probe foo-2-private >>>>>>> Probe successful >>>>>>> [root at localhost ~]# gluster peer status >>>>>>> Number of Peers: 1 >>>>>>> >>>>>>> Hostname: foo-2-private >>>>>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3 >>>>>>> State: Peer in Cluster (Connected) >>>>>>> [root at localhost ~]# gluster volume create foo >>>>>>> foo-1-private:/mnt/brick >>>>>>> Creation of volume foo has been successful. Please start the >>>>>>> volume to >>>>>>> access data. >>>>>>> [root at localhost ~]# gluster volume start foo >>>>>>> Starting volume foo has been successful >>>>>>> [root at localhost ~]# gluster volume add-brick foo >>>>>>> foo-2-private:/mnt/brick >>>>>>> Add Brick successful >>>>>>> [root at localhost ~]# gluster peer probe foo-3-private >>>>>>> Probe successful >>>>>>> [root at localhost ~]# gluster peer status >>>>>>> Number of Peers: 2 >>>>>>> >>>>>>> Hostname: foo-2-private >>>>>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3 >>>>>>> State: Peer in Cluster (Connected) >>>>>>> >>>>>>> Hostname: foo-3-private >>>>>>> Uuid: 7587ae34-9209-484a-9576-3939e061720c >>>>>>> State: Peer Rejected (Connected) >>>>>>> [root at localhost ~]# exit >>>>>>> logout >>>>>>> Connection to foo-1-private closed. >>>>>>> [root at vhead-010 ~]# find foo_log_and_conf >>>>>>> foo_log_and_conf >>>>>>> foo_log_and_conf/foo-2-private >>>>>>> foo_log_and_conf/foo-2-private/glusterd >>>>>>> foo_log_and_conf/foo-2-private/glusterd/glusterd.info >>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs >>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/nfs-server.vol >>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run >>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run/nfs.pid >>>>>>> foo_log_and_conf/foo-2-private/glusterd/peers >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722 >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/info >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/cksum >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run/foo-2-private-mnt-brick.pid >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo-fuse.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-2-private/glusterfs >>>>>>> foo_log_and_conf/foo-2-private/glusterfs/nfs.log >>>>>>> foo_log_and_conf/foo-2-private/glusterfs/bricks >>>>>>> foo_log_and_conf/foo-2-private/glusterfs/bricks/mnt-brick.log >>>>>>> foo_log_and_conf/foo-2-private/glusterfs/.cmd_log_history >>>>>>> foo_log_and_conf/foo-2-private/glusterfs/etc-glusterfs-glusterd.vol.log >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private >>>>>>> foo_log_and_conf/foo-1-private/glusterd >>>>>>> foo_log_and_conf/foo-1-private/glusterd/glusterd.info >>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs >>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/nfs-server.vol >>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run >>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run/nfs.pid >>>>>>> foo_log_and_conf/foo-1-private/glusterd/peers >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/peers/20b73d9a-ede0-454f-9fbb-b0eee9ce26a3 >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/peers/7587ae34-9209-484a-9576-3939e061720c >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/info >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/cksum >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run/foo-1-private-mnt-brick.pid >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo-fuse.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-1-private/glusterfs >>>>>>> foo_log_and_conf/foo-1-private/glusterfs/nfs.log >>>>>>> foo_log_and_conf/foo-1-private/glusterfs/bricks >>>>>>> foo_log_and_conf/foo-1-private/glusterfs/bricks/mnt-brick.log >>>>>>> foo_log_and_conf/foo-1-private/glusterfs/.cmd_log_history >>>>>>> foo_log_and_conf/foo-1-private/glusterfs/etc-glusterfs-glusterd.vol.log >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private >>>>>>> foo_log_and_conf/foo-3-private/glusterd >>>>>>> foo_log_and_conf/foo-3-private/glusterd/glusterd.info >>>>>>> foo_log_and_conf/foo-3-private/glusterd/nfs >>>>>>> foo_log_and_conf/foo-3-private/glusterd/nfs/run >>>>>>> foo_log_and_conf/foo-3-private/glusterd/peers >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722 >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/info >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick >>>>>>> >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/cksum >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo-fuse.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol >>>>>>> >>>>>>> foo_log_and_conf/foo-3-private/glusterfs >>>>>>> foo_log_and_conf/foo-3-private/glusterfs/nfs.log >>>>>>> foo_log_and_conf/foo-3-private/glusterfs/bricks >>>>>>> foo_log_and_conf/foo-3-private/glusterfs/bricks/mnt-brick.log >>>>>>> foo_log_and_conf/foo-3-private/glusterfs/.cmd_log_history >>>>>>> foo_log_and_conf/foo-3-private/glusterfs/etc-glusterfs-glusterd.vol.log >>>>>>> >>>>>>> [root at vhead-010 ~]# exit >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> (2011/08/16 9:35), Mohit Anchlia wrote: >>>>>>>> >>>>>>>> I should have also asked you to stop and delete volume before >>>>>>>> getting >>>>>>>> rid of gluster config files. Can you get rid of directories also >>>>>>>> inside vols and try to restart? It's trying to look for volume >>>>>>>> files >>>>>>>> that we just removed. >>>>>>>> >>>>>>>> Also, just disable iptables for now explicitly. >>>>>>>> >>>>>>>> On Mon, Aug 15, 2011 at 5:22 PM, Tomoaki Sato<tsato at valinux.co.jp> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> 1) run peer detach for all the servers >>>>>>>>> >>>>>>>>> done. >>>>>>>>> >>>>>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and >>>>>>>>>> iptables >>>>>>>>>> are turned off. >>>>>>>>> >>>>>>>>> done. >>>>>>>>> by the way, the same test on 3.1.5-1 works fine with same >>>>>>>>> environment. >>>>>>>>> >>>>>>>>>> 3) remove config files under /etc/glusterd >>>>>>>>> >>>>>>>>> please review following logs. >>>>>>>>> >>>>>>>>>> 4) run your tests again. >>>>>>>>> >>>>>>>>> I don't know why but glusterd service failed to start on all 3 >>>>>>>>> hosts. >>>>>>>>> >>>>>>>>> [root at vhead-010 ~]# date >>>>>>>>> Tue Aug 16 09:12:53 JST 2011 >>>>>>>>> [root at vhead-010 ~]# cat a.sh >>>>>>>>> #!/bin/bash >>>>>>>>> for i in foo-{1..3}-private >>>>>>>>> do >>>>>>>>> ssh ${i} service glusterd stop >>>>>>>>> ssh ${i} 'find /etc/glusterd -type f|xargs rm -f' >>>>>>>>> ssh ${i} service iptables restart >>>>>>>>> ssh ${i} iptables -vL >>>>>>>>> ssh ${i} service glusterd start >>>>>>>>> ssh ${i} find /etc/glusterd >>>>>>>>> ssh ${i} service glusterd status >>>>>>>>> done >>>>>>>>> [root at vhead-010 ~]# bash a.sh >>>>>>>>> Stopping glusterd:[ OK ] >>>>>>>>> Flushing firewall rules: [ OK ] >>>>>>>>> Setting chains to policy ACCEPT: filter [ OK ] >>>>>>>>> Unloading iptables modules: [ OK ] >>>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> >>>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> >>>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> Starting glusterd:[ OK ] >>>>>>>>> /etc/glusterd >>>>>>>>> /etc/glusterd/glusterd.info >>>>>>>>> /etc/glusterd/nfs >>>>>>>>> /etc/glusterd/nfs/run >>>>>>>>> /etc/glusterd/peers >>>>>>>>> /etc/glusterd/vols >>>>>>>>> /etc/glusterd/vols/foo >>>>>>>>> /etc/glusterd/vols/foo/bricks >>>>>>>>> /etc/glusterd/vols/foo/run >>>>>>>>> glusterd is stopped >>>>>>>>> Stopping glusterd:[ OK ] >>>>>>>>> Flushing firewall rules: [ OK ] >>>>>>>>> Setting chains to policy ACCEPT: filter [ OK ] >>>>>>>>> Unloading iptables modules: [ OK ] >>>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> >>>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> >>>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> Starting glusterd:[ OK ] >>>>>>>>> /etc/glusterd >>>>>>>>> /etc/glusterd/glusterd.info >>>>>>>>> /etc/glusterd/nfs >>>>>>>>> /etc/glusterd/nfs/run >>>>>>>>> /etc/glusterd/peers >>>>>>>>> /etc/glusterd/vols >>>>>>>>> /etc/glusterd/vols/foo >>>>>>>>> /etc/glusterd/vols/foo/bricks >>>>>>>>> /etc/glusterd/vols/foo/run >>>>>>>>> glusterd is stopped >>>>>>>>> Stopping glusterd:[ OK ] >>>>>>>>> Flushing firewall rules: [ OK ] >>>>>>>>> Setting chains to policy ACCEPT: filter [ OK ] >>>>>>>>> Unloading iptables modules: [ OK ] >>>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> >>>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> >>>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) >>>>>>>>> pkts bytes target prot opt in out source >>>>>>>>> destination >>>>>>>>> Starting glusterd:[ OK ] >>>>>>>>> /etc/glusterd >>>>>>>>> /etc/glusterd/glusterd.info >>>>>>>>> /etc/glusterd/nfs >>>>>>>>> /etc/glusterd/nfs/run >>>>>>>>> /etc/glusterd/peers >>>>>>>>> /etc/glusterd/vols >>>>>>>>> /etc/glusterd/vols/foo >>>>>>>>> /etc/glusterd/vols/foo/bricks >>>>>>>>> /etc/glusterd/vols/foo/run >>>>>>>>> glusterd is stopped >>>>>>>>> [root at vhead-010 ~]# date >>>>>>>>> Tue Aug 16 09:13:20 JST 2011 >>>>>>>>> [root at vhead-010 ~]# ssh foo-1-private >>>>>>>>> Last login: Tue Aug 16 09:06:57 2011 from dlp.local.valinux.co.jp >>>>>>>>> [root at localhost ~]# tail -20 >>>>>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log >>>>>>>>> ... >>>>>>>>> [2011-08-16 09:13:01.85858] I [glusterd.c:304:init] >>>>>>>>> 0-management: Using >>>>>>>>> /etc/glusterd as working directory >>>>>>>>> [2011-08-16 09:13:01.87294] E >>>>>>>>> [rpc-transport.c:799:rpc_transport_load] >>>>>>>>> 0-rpc-transport: >>>>>>>>> /opt/glusterfs/3.1.6/lib64/glusterfs/3.1.6/rpc-transport/rdma.so: >>>>>>>>> cannot >>>>>>>>> open shared object file: No such file or directory >>>>>>>>> [2011-08-16 09:13:01.87340] E >>>>>>>>> [rpc-transport.c:803:rpc_transport_load] >>>>>>>>> 0-rpc-transport: volume 'rdma.management': transport-type >>>>>>>>> 'rdma' is not >>>>>>>>> valid or not found on this machine >>>>>>>>> [2011-08-16 09:13:01.87402] E >>>>>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd: >>>>>>>>> Unable >>>>>>>>> to >>>>>>>>> retrieve store handle for /etc/glusterd/glusterd.info, error: >>>>>>>>> No such >>>>>>>>> file >>>>>>>>> or directory >>>>>>>>> [2011-08-16 09:13:01.87422] E >>>>>>>>> [glusterd-store.c:761:glusterd_retrieve_uuid] >>>>>>>>> 0-: Unable to get store handle! >>>>>>>>> [2011-08-16 09:13:01.87514] I [glusterd.c:95:glusterd_uuid_init] >>>>>>>>> 0-glusterd: >>>>>>>>> generated UUID: c0cef9f9-a79e-4189-8955-d83927db9cee >>>>>>>>> [2011-08-16 09:13:01.87681] E >>>>>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd: >>>>>>>>> Unable >>>>>>>>> to >>>>>>>>> retrieve store handle for /etc/glusterd/vols/foo/info, error: >>>>>>>>> No such >>>>>>>>> file >>>>>>>>> or directory >>>>>>>>> [2011-08-16 09:13:01.87704] E >>>>>>>>> [glusterd-store.c:1328:glusterd_store_retrieve_volumes] 0-: >>>>>>>>> Unable to >>>>>>>>> restore volume: foo >>>>>>>> >>>>>>>>> [2011-08-16 09:13:01.87732] E [xlator.c:843:xlator_init] >>>>>>>>> 0-management: >>>>>>>>> Initialization of volume 'management' failed, review your >>>>>>>>> volfile again >>>>>>>>> [2011-08-16 09:13:01.87751] E [graph.c:331:glusterfs_graph_init] >>>>>>>>> 0-management: initializing translator failed >>>>>>>>> [2011-08-16 09:13:01.87818] I [glusterfsd.c:712:cleanup_and_exit] >>>>>>>>> 0-glusterfsd: shutting down >>>>>>>>> [root at localhost ~]# exit >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> (2011/08/16 8:52), Mohit Anchlia wrote: >>>>>>>>>> >>>>>>>>>> Logs are generally in /var/log/gluster >>>>>>>>>> >>>>>>>>>> Since you are playing with it. I would suggest this: >>>>>>>>>> >>>>>>>>>> 1) run peer detach for all the servers >>>>>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and >>>>>>>>>> iptables >>>>>>>>>> are turned off. >>>>>>>>>> 3) remove config files under /etc/glusterd >>>>>>>>>> 4) run your tests again. >>>>>>>>>> >>>>>>>>>> On Mon, Aug 15, 2011 at 4:28 PM, Tomoaki >>>>>>>>>> Sato<tsato at valinux.co.jp> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Thanks, Mohit >>>>>>>>>>> >>>>>>>>>>> (2011/08/16 8:05), Mohit Anchlia wrote: >>>>>>>>>>>> >>>>>>>>>>>> What's in your logs? >>>>>>>>>>> >>>>>>>>>>> I can obtain logs needed. could you tell me the instruction >>>>>>>>>>> to take >>>>>>>>>>> the >>>>>>>>>>> logs? >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Did you have foo-3-private before in your gluster cluster >>>>>>>>>>>> ever or >>>>>>>>>>>> adding this host for the first time? >>>>>>>>>>> >>>>>>>>>>> It was first time. >>>>>>>>>>> All foo-X-private has no entries in /etc/glusterd/peers/ and >>>>>>>>>>> /etc/glusterd/vols/. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Try gluster peer detach and then remove any left over >>>>>>>>>>>> configuration >>>>>>>>>>>> in >>>>>>>>>>>> /etc/glusterd config directory. After that try again and >>>>>>>>>>>> see if that >>>>>>>>>>>> works. >>>>>>>>>>> >>>>>>>>>>> [root at vhead-010 ~]# date >>>>>>>>>>> Tue Aug 16 08:17:49 JST 2011 >>>>>>>>>>> [root at vhead-010 ~]# cat a.sh >>>>>>>>>>> #!/bin/bash >>>>>>>>>>> for i in foo-{1..3}-private >>>>>>>>>>> do >>>>>>>>>>> ssh ${i} service glusterd stop >>>>>>>>>>> ssh ${i} rm -rf /etc/glusterd/peers/* >>>>>>>>>>> ssh ${i} rm -rf /etc/glusterd/vols/* >>>>>>>>>>> ssh ${i} service glusterd start >>>>>>>>>>> ssh ${i} find /etc/glusterd >>>>>>>>>>> done >>>>>>>>>>> [root at vhead-010 ~]# bash a.sh >>>>>>>>>>> Stopping glusterd:[ OK ] >>>>>>>>>>> Starting glusterd:[ OK ] >>>>>>>>>>> /etc/glusterd >>>>>>>>>>> /etc/glusterd/glusterd.info >>>>>>>>>>> /etc/glusterd/nfs >>>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol >>>>>>>>>>> /etc/glusterd/nfs/run >>>>>>>>>>> /etc/glusterd/peers >>>>>>>>>>> /etc/glusterd/vols >>>>>>>>>>> Stopping glusterd:[ OK ] >>>>>>>>>>> Starting glusterd:[ OK ] >>>>>>>>>>> /etc/glusterd >>>>>>>>>>> /etc/glusterd/glusterd.info >>>>>>>>>>> /etc/glusterd/nfs >>>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol >>>>>>>>>>> /etc/glusterd/nfs/run >>>>>>>>>>> /etc/glusterd/peers >>>>>>>>>>> /etc/glusterd/vols >>>>>>>>>>> Stopping glusterd:[ OK ] >>>>>>>>>>> Starting glusterd:[ OK ] >>>>>>>>>>> /etc/glusterd >>>>>>>>>>> /etc/glusterd/glusterd.info >>>>>>>>>>> /etc/glusterd/nfs >>>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol >>>>>>>>>>> /etc/glusterd/nfs/run >>>>>>>>>>> /etc/glusterd/peers >>>>>>>>>>> /etc/glusterd/vols >>>>>>>>>>> [root at vhead-010 ~]# ssh foo-1-private >>>>>>>>>>> [root at localhost ~]# gluster peer probe foo-2-private >>>>>>>>>>> Probe successful >>>>>>>>>>> [root at localhost ~]# gluster peer status >>>>>>>>>>> Number of Peers: 1 >>>>>>>>>>> >>>>>>>>>>> Hostname: foo-2-private >>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2 >>>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>>> [root at localhost ~]# gluster volume create foo >>>>>>>>>>> foo-1-private:/mnt/brick >>>>>>>>>>> Creation of volume foo has been successful. Please start the >>>>>>>>>>> volume >>>>>>>>>>> to >>>>>>>>>>> access da >>>>>>>>>>> ta. >>>>>>>>>>> [root at localhost ~]# gluster volume start foo >>>>>>>>>>> Starting volume foo has been successful >>>>>>>>>>> [root at localhost ~]# gluster volume add-brick foo >>>>>>>>>>> foo-2-private:/mnt/brick >>>>>>>>>>> Add Brick successful >>>>>>>>>>> [root at localhost ~]# gluster peer probe foo-3-private >>>>>>>>>>> Probe successful >>>>>>>>>>> [root at localhost ~]# gluster peer status >>>>>>>>>>> Number of Peers: 2 >>>>>>>>>>> >>>>>>>>>>> Hostname: foo-2-private >>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2 >>>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>>> >>>>>>>>>>> Hostname: foo-3-private >>>>>>>>>>> Uuid: 7fb98dac-fef7-4b33-837c-6483a767ec3e >>>>>>>>>>> State: Peer Rejected (Connected) >>>>>>>>>>> [root at localhost ~]# cat /var/log/glusterfs/.cmd_log_history >>>>>>>>>>> ... >>>>>>>>>>> [2011-08-16 08:20:28.862619] peer probe : on host >>>>>>>>>>> foo-2-private:24007 >>>>>>>>>>> [2011-08-16 08:20:28.912419] peer probe : on host >>>>>>>>>>> foo-2-private:24007 >>>>>>>>>>> FAILED >>>>>>>>>>> [2011-08-16 08:20:58.382350] Volume create : on volname: foo >>>>>>>>>>> attempted >>>>>>>>>>> [2011-08-16 08:20:58.382461] Volume create : on volname: foo >>>>>>>>>>> type:DEFAULT >>>>>>>>>>> count: >>>>>>>>>>> 1 bricks: foo-1-private:/mnt/brick >>>>>>>>>>> [2011-08-16 08:20:58.384674] Volume create : on volname: foo >>>>>>>>>>> SUCCESS >>>>>>>>>>> [2011-08-16 08:21:04.831772] volume start : on volname: foo >>>>>>>>>>> SUCCESS >>>>>>>>>>> [2011-08-16 08:21:22.682292] Volume add-brick : on volname: foo >>>>>>>>>>> attempted >>>>>>>>>>> [2011-08-16 08:21:22.682385] Volume add-brick : volname: foo >>>>>>>>>>> type >>>>>>>>>>> DEFAULT >>>>>>>>>>> count: >>>>>>>>>>> 1 bricks: foo-2-private:/mnt/brick >>>>>>>>>>> [2011-08-16 08:21:22.682499] Volume add-brick : on volname: foo >>>>>>>>>>> SUCCESS >>>>>>>>>>> [2011-08-16 08:21:39.124574] peer probe : on host >>>>>>>>>>> foo-3-private:24007 >>>>>>>>>>> [2011-08-16 08:21:39.135609] peer probe : on host >>>>>>>>>>> foo-3-private:24007 >>>>>>>>>>> FAILED >>>>>>>>>>> >>>>>>>>>>> Tomo >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 15, 2011 at 3:37 PM, Tomoaki >>>>>>>>>>>> Sato<tsato at valinux.co.jp> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> following instructions work fine with 3.1.5-1 but with >>>>>>>>>>>>> 3.1.6-1. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. make a new file system without peers. [OK] >>>>>>>>>>>>> >>>>>>>>>>>>> foo-1-private# gluster volume create foo >>>>>>>>>>>>> foo-1-private:/mnt/brick >>>>>>>>>>>>> foo-1-private# gluster volume start foo >>>>>>>>>>>>> foo-1-private# gluster peer status >>>>>>>>>>>>> No peers present >>>>>>>>>>>>> foo-1-private# >>>>>>>>>>>>> >>>>>>>>>>>>> 2. add a peer to the file system. [NG] >>>>>>>>>>>>> >>>>>>>>>>>>> foo-1-private# gluster peer probe foo-2-private >>>>>>>>>>>>> Probe successful >>>>>>>>>>>>> foo-1-private# gluster peer status >>>>>>>>>>>>> Number of Peers: 1 >>>>>>>>>>>>> >>>>>>>>>>>>> Hostname: foo-2-private >>>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2 >>>>>>>>>>>>> State: Peer Rejected (Connected) >>>>>>>>>>>>> foo-1-private# gluster volume add-brick foo >>>>>>>>>>>>> foo-2-private:/mnt/brick >>>>>>>>>>>>> Host foo-2-private not connected >>>>>>>>>>>>> foo-1-private# >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> following instructions work fine even with 3.1.6-1. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. make a new file system with single peer. [OK] >>>>>>>>>>>>> >>>>>>>>>>>>> foo-1-private# gluster peer status >>>>>>>>>>>>> No peer presents >>>>>>>>>>>>> foo-1-private# gluster peer probe foo-2-private >>>>>>>>>>>>> Probe successful >>>>>>>>>>>>> foo-1-private# gluster peer status >>>>>>>>>>>>> Number of Peers: 1 >>>>>>>>>>>>> >>>>>>>>>>>>> Hostname: foo-2-private >>>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2 >>>>>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>>>>> foo-1-private# gluster volume create foo >>>>>>>>>>>>> foo-1-private:/mnt/brick >>>>>>>>>>>>> Creation of volume foo has been successful. Please start >>>>>>>>>>>>> the volume >>>>>>>>>>>>> to >>>>>>>>>>>>> access data. >>>>>>>>>>>>> foo-1-private# gluster volume start foo >>>>>>>>>>>>> Starting volume foo has been successful >>>>>>>>>>>>> foo-1-private# gluster volume add-brick foo >>>>>>>>>>>>> foo-2-private:/mnt/brick >>>>>>>>>>>>> Add Brick successful >>>>>>>>>>>>> foo-1-private# >>>>>>>>>>>>> >>>>>>>>>>>>> But ... >>>>>>>>>>>>> >>>>>>>>>>>>> 2. add a peer to the file system. [NG] >>>>>>>>>>>>> >>>>>>>>>>>>> foo-1-private# gluster peer probe foo-3-private >>>>>>>>>>>>> Probe successful >>>>>>>>>>>>> foo-1-private# gluster peer status >>>>>>>>>>>>> Number of Peers: 2 >>>>>>>>>>>>> >>>>>>>>>>>>> Hostname: foo-2-private >>>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2 >>>>>>>>>>>>> State: Peer in Cluster (Connected) >>>>>>>>>>>>> >>>>>>>>>>>>> Hostname: foo-3-private >>>>>>>>>>>>> Uuid: 7fb98dac-fef704b33-837c-6483a767ec3e >>>>>>>>>>>>> State: Peer Rejected (Connected) >>>>>>>>>>>>> foo-1-private# gluster volume add-brick foo >>>>>>>>>>>>> foo-3-private:/mnt/brick >>>>>>>>>>>>> Host foo-3-private not connected >>>>>>>>>>>>> foo-1-private# >>>>>>>>>>>>> >>>>>>>>>>>>> How should I add extra peers to existing file systems ? >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110817/7e62a6d9/attachment-0001.htm>
