Tom- That looks like the same problem I had after upgrading from GlusterFS 3.2.6 to 3.3. Here is a link to a thread that includes the diagnosis (missing username and password entries in vol files) and solution suggested by Rahul Hinduja. http://gluster.org/pipermail/gluster-users/2012-August/011055.html -Dan. On 09/15/2012 02:32 AM, gluster-users-request at gluster.org wrote: > Message: 2 > Date: Fri, 14 Sep 2012 20:57:25 +0000 > From: Thomas Indelli<tindelli at uchicago.edu> > Subject: Problems adding new server to pool > To: "gluster-users at gluster.org"<gluster-users at gluster.org> > Message-ID: > <A4C9147B95CDDC4097015BC5551AEB6E0473AE80 at xm-mbx-01-prod.ad.uchicago.edu> > > Content-Type: text/plain; charset="us-ascii" > > Hello, > > I don't have a great deal of experience yet with Gluster, and I'm having some tremendous difficulties adding an additional (3rd) server to the pool I have set up. This is running Gluster 3.3.0 on RHEL 6.3, using RPMs: > # rpm -qa | grep gluster > glusterfs-rdma-3.3.0-1.el6.x86_64 > glusterfs-server-3.3.0-1.el6.x86_64 > glusterfs-3.3.0-1.el6.x86_64 > glusterfs-fuse-3.3.0-1.el6.x86_64 > > When I first set it up, I was able to add the second server without an issue using gluster peer probe vothgluster2, however when I do the same here, it just hangs for a couple minutes, then returns to the prompt without successfully probing the server. >> From the first server: > # gluster volume info > > Volume Name: home > Type: Distribute > Volume ID: a6f62691-a876-431a-a941-d3b76e197eeb > Status: Started > Number of Bricks: 4 > Transport-type: tcp,rdma > Bricks: > Brick1: vothgluster1:/export/brick1 > Brick2: vothgluster1:/export/brick2 > Brick3: vothgluster2:/export/brick1 > Brick4: vothgluster2:/export/brick2 > Options Reconfigured: > nfs.rpc-auth-allow: 128.135.10.*,205.208.111.*,128.135.233.*,128.135.186.*,10.135.10.*,205.208.76.203 > nfs.disable: on > nfs.register-with-portmap: on > performance.io-cache: on > nfs.port: 2049 > features.quota: off > performance.cache-size: 64MB > performance.read-ahead: off > performance.write-behind-window-size: 1MB > nfs.ports-insecure: on > > # gluster peer status > Number of Peers: 2 > > Hostname: vothgluster2 > Uuid: 44033fcb-f6da-4491-b860-d05bb3ce23f7 > State: Peer in Cluster (Connected) > > Hostname: vothgluster3 > Uuid: 924322ef-5e55-427b-a393-90903ccb7d16 > State: Establishing Connection (Connected) >> From the second server, gluster volume info reports the same. > # gluster peer status > Number of Peers: 1 > > Hostname: 205.208.111.14 > Uuid: 70ee8eed-224e-4031-8cfe-52004c5ccf47 > State: Peer in Cluster (Connected) >> From the new (problematic) server: > # gluster volume info > No volumes present > > # gluster peer status > Number of Peers: 1 > > Hostname: 205.208.111.14 > Uuid: 00000000-0000-0000-0000-000000000000 > State: Connected to Peer (Connected) > > Forward and Reverse DNS lookups between all 3 servers work fine. Ping works fine. IB connectivity works fine (using tcp,rdma transport type). > > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on vothgluster3 > [2012-09-14 15:16:27.528532] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.3.0 > [2012-09-14 15:16:27.530778] I [glusterd.c:807:init] 0-management: Using /var/lib/glusterd as working directory > [2012-09-14 15:16:28.088422] I [glusterd.c:95:glusterd_uuid_init] 0-glusterd: retrieved UUID: 924322ef-5e55-427b-a393-90903ccb7d16 > [2012-09-14 15:16:28.097455] I [glusterd.c:265:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system > Given volfile: > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option working-directory /var/lib/glusterd > 4: option transport-type socket,rdma > 5: option transport.socket.keepalive-time 10 > 6: option transport.socket.keepalive-interval 2 > 7: option transport.socket.read-fail-log off > 8: end-volume > > +------------------------------------------------------------------------------+ > [2012-09-14 15:16:32.760625] I [glusterd-handler.c:1753:glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 70ee8eed-224e-4031-8cfe-52004c5ccf47 > [2012-09-14 15:16:32.760707] I [glusterd-handler.c:411:glusterd_friend_find] 0-glusterd: Unable to find peer by uuid > [2012-09-14 15:16:32.760929] I [glusterd-handler.c:423:glusterd_friend_find] 0-glusterd: Unable to find hostname: 205.208.111.14 > [2012-09-14 15:16:32.760950] I [glusterd-handler.c:1767:glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 205.208.111.14 (24007) > [2012-09-14 15:16:32.766007] I [glusterd-handler.c:2204:glusterd_friend_add] 0-management: connect returned 0 > [2012-09-14 15:16:32.766067] I [glusterd-handler.c:1793:glusterd_handle_probe_query] 0-glusterd: Responded to 205.208.111.14, op_ret: 0, op_errno: 0, ret: 0 > [2012-09-14 15:16:32.766572] I [glusterd-handshake.c:397:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd mgmt, Num (1238433), Version (2) > [2012-09-14 15:16:32.766620] I [glusterd-handshake.c:403:glusterd_set_clnt_mgmt_program] 0-: Using Program Peer mgmt, Num (1238437), Version (2) > [2012-09-14 15:18:44.916221] I [glusterd-handler.c:813:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req > [2012-09-14 15:19:41.020394] I [glusterd-handler.c:860:glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req > [2012-09-14 15:22:40.807457] I [glusterd-handler.c:860:glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req > [2012-09-14 15:22:48.655202] I [glusterd-handler.c:813:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req > > ----- > > Please let me know any other information to provide. > > Thanks in advance, > Tom > > > > -- > Tom Indelli > tindelli at uchicago.edu<mailto:tindelli at uchicago.edu> > Systems Administrator > Physical Sciences Division > The University of Chicago
