Other people have seen these too. Seems like some kind of deadlock issue. What commands did you run prior to running rebalance command? Can you restart /etc/init.d/glusterd on all the servers and try again? On Tue, May 24, 2011 at 10:02 AM, Karim Latouche <klatouche at digitalchocolate.com> wrote: > Hi > > Sorry i forgot to include the log message I got when running a > gluster volume rebalance > >> I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] >> 0-glusterd: Received rebalance volume 1 on dccbackup >> [2011-05-24 03:59:23.645067] W >> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading >> from socket failed. Error (Transport endpoint is not connected), peer >> (127.0.0.1:1019) >> [2011-05-24 03:59:28.323938] I >> [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 0-glusterd: >> Received rebalance volume 1 on dccbackup >> [2011-05-24 03:59:28.352874] W >> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading >> from socket failed. Error (Transport endpoint is not connected), peer >> (127.0.0.1:1018) > > And this is the output of the command > > gluster volume rebalance dccbackup start > starting rebalance on volume dccbackup has been unsuccessful > > Thx > > KL > >> Look in the gluster logs and see what it says. /var/log/gluster* >> >> If you intention is to migrate data then alternatively you can also do: >> >> gluster volume rebalance VOLNAME start >> >> >> On Tue, May 24, 2011 at 2:59 AM, Karim Latouche >> <klatouche at digitalchocolate.com> ?wrote: >>> >>> Hi >>> >>> Thx for your help. >>> >>> I finaly could add the last brick . >>> >>> I now have a Distributed-replicated volume >>> >>>> gluster volume info >>>> >>>> Volume Name: dccbackup >>>> Type: Distributed-Replicate >>>> Status: Started >>>> Number of Bricks: 2 x 2 = 4 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: mysqldccbackup:/export >>>> Brick2: mysqldccbackupmirror:/export >>>> Brick3: gfsbrick1:/brick1 >>>> Brick4: gfsbrick2:/brick2 >>> >>> On the client side though ?I don't see any increasing in storage size. >>> I tried to rebalance >>> >>> gluster volume rebalance ?dccbackup fix-layout start >>> >>> and got . >>> >>>> starting rebalance on volume dccbackup has been unsuccessful >>> >>> I'm kind of stuck here. >>> >>> Thank you >>> >>> KL >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>> What is the output of: >>>> >>>> # ls /etc/glusterd/peers >>>> 39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> ?4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> ?a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> >>>> Above output is mine. >>>> >>>> If you cat each of these files, there should be one for each peer OTHER >>>> than the host that your are logged into), and hostnames each ?files, >>>> like >>>> this: >>>> # cat /etc/glusterd/peers/* >>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> state=3 >>>> hostname1=jc1letgfs15-pfs1 >>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> state=3 >>>> hostname1=jc1letgfs18-pfs1 >>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> state=3 >>>> hostname1=jc1letgfs14-pfs1 >>>> >>>> Here is a listing of all the peer files on all four nodes of my >>>> GlusterFS >>>> servers - I created this configuration in the same way you did - two >>>> servers, then add two servers: >>>> >>>> jc1letgfs14 >>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> state=3 >>>> hostname1=jc1letgfs15-pfs1 >>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> state=3 >>>> hostname1=jc1letgfs18-pfs1 >>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>> state=3 >>>> hostname1=10.20.72.191 >>>> >>>> jc1letgfs15 >>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> state=3 >>>> hostname1=jc1letgfs18-pfs1 >>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>> state=3 >>>> hostname1=10.20.72.191 >>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> state=3 >>>> hostname1=jc1letgfs14-pfs1 >>>> >>>> jc1letgfs17 >>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> state=3 >>>> hostname1=jc1letgfs15-pfs1 >>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>> state=3 >>>> hostname1=jc1letgfs18-pfs1 >>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> state=3 >>>> hostname1=jc1letgfs14-pfs1 >>>> >>>> jc1letgfs18 >>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>> state=3 >>>> hostname1=jc1letgfs15-pfs1 >>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>> state=3 >>>> hostname1=10.20.72.191 >>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>> state=3 >>>> hostname1=jc1letgfs14-pfs1 >>>> >>>> I hope this is of some help. >>>> >>>> James Burnash >>>> Unix Engineer >>>> >>>> >>>> -----Original Message----- >>>> From: gluster-users-bounces at gluster.org >>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Karim Latouche >>>> Sent: Monday, May 23, 2011 1:02 PM >>>> To: Mohit Anchlia >>>> Cc: gluster-users at gluster.org >>>> Subject: Re: Trouble to add brick >>>> >>>> hi >>>> >>>> Here is the output of gluster peer status ?command of the four nodes >>>> >>>> mysqldccbackup mysqldccbackupmirror are the two first nodes >>>> >>>> I got issues when I tried to add gfsbrick1 and two to the already >>>> working >>>> cluster >>>> >>>> >>>> >>>>> [root at gfsbrick1 ~]# gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: 10.0.4.225 >>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>> State: Establishing Connection (Connected) >>>>> [root at gfsbrick2 ~]# gluster peer status >>>>> Number of Peers: 2 >>>>> >>>>> Hostname: 10.0.4.225 >>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: mysqldccbackupmirror >>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939 >>>>> State: Peer Rejected (Connected) >>>>> [root at mysqldccbackup ~]# gluster peer status >>>>> Number of Peers: 3 >>>>> >>>>> Hostname: mysqldccbackupmirror >>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: gfsbrick2 >>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: gfsbrick1 >>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>> State: Establishing Connection (Connected) >>>> >>>>> root at mysqldccbackupmirror ~]# gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: 10.0.4.225 >>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6 >>>>> State: Peer in Cluster (Connected) >>>>> [root at mysqldccbackupmirror ~]# gluster peer status >>>>> Number of Peers: 1 >>>>> >>>>> Hostname: 10.0.4.225 >>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6 >>>>> State: Peer in Cluster (Connected) >>>> >>>> Thx a lot >>>> >>>> KL >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> Please send the complete output of gluster peer status from both the >>>>> nodes. >>>>> >>>>> On Mon, May 23, 2011 at 9:44 AM, Karim Latouche >>>>> <klatouche at digitalchocolate.com> ? ? wrote: >>>>>> >>>>>> hi >>>>>> >>>>>> Thx for your answer. >>>>>> >>>>>> I should have mention that I already set up a proper hosts file to >>>>>> avoid >>>>>> DNS >>>>>> resolution issue but I still have same issue. >>>>>> >>>>>> The last brick is allways the one having issues to be added . I tried >>>>>> adding >>>>>> gfsbrick2 first then gfsbrick1 and the result is the same >>>>>> >>>>>> I get >>>>>> >>>>>>> Hostname: gfsbrick2 >>>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f >>>>>>> State: Peer in Cluster (Connected) >>>>>>> >>>>>>> Hostname: gfsbrick1 >>>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>>> State: Establishing Connection (Connected) >>>>>> >>>>>> instead of >>>>>> >>>>>> >>>>>>> Hostname: gfsbrick2 >>>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>>> State: Establishing Connection (Connected) >>>>>> >>>>>> It 's really strange. >>>>>> >>>>>> >>>>>> Thx >>>>>> >>>>>> KL >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Karim, >>>>>>> ? ? ?gfsbrick1 is not able to contact gfsbrick2 yet. Generally >>>>>>> happens >>>>>>> if >>>>>>> the DNS resolutions dont happen as expected. You can map them >>>>>>> manually >>>>>>> in >>>>>>> /etc/hosts for both the machines?. >>>>>>> >>>>>>> something like: >>>>>>> on gfsbrick1 >>>>>>> <ip-of-brick2> ? ? ? gfsbrick2 >>>>>>> on gfsbrick2 >>>>>>> <ip-of-brick1> ? ? ? gfsbrick1 >>>>>>> >>>>>>> Pranith. >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Karim Latouche"<klatouche at digitalchocolate.com> >>>>>>> To: gluster-users at gluster.org >>>>>>> Sent: Monday, May 23, 2011 7:56:22 PM >>>>>>> Subject: Trouble to add brick >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I got a strange problems >>>>>>> >>>>>>> I run a two bricks replicated volume and I tried to add 2 more bricks >>>>>>> >>>>>>> The first one went ok >>>>>>> >>>>>>>> gluster peer probe gfsbrick1 >>>>>>>> Probe successful >>>>>>> >>>>>>> It seems I can't add the second one >>>>>>> >>>>>>> I get a >>>>>>> >>>>>>>> Hostname: gfsbrick2 >>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>>>> State: Establishing Connection (Connected) >>>>>>> >>>>>>> Did anyone encounter this issue ? >>>>>>> >>>>>>> Thx >>>>>>> >>>>>>> KL >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>> >>>> >>>> DISCLAIMER: >>>> This e-mail, and any attachments thereto, is intended only for use by >>>> the >>>> addressee(s) named herein and may contain legally privileged and/or >>>> confidential information. If you are not the intended recipient of this >>>> e-mail, you are hereby notified that any dissemination, distribution or >>>> copying of this e-mail, and any attachments thereto, is strictly >>>> prohibited. >>>> If you have received this in error, please immediately notify me and >>>> permanently delete the original and any copy of any e-mail and any >>>> printout >>>> thereof. E-mail transmission cannot be guaranteed to be secure or >>>> error-free. The sender therefore does not accept liability for any >>>> errors or >>>> omissions in the contents of this message which arise as a result of >>>> e-mail >>>> transmission. >>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, >>>> at >>>> its discretion, monitor and review the content of all e-mail >>>> communications. >>>> http://www.knight.com >>> > >
