On 5/24/11 8:09 PM, Mohit Anchlia wrote: > Other people have seen these too. Seems like some kind of deadlock > issue. What commands did you run prior to running rebalance command? I tried gluster volume rebalance dccbackup fix-layout start but was unsuccessfull > Can you restart /etc/init.d/glusterd on all the servers and try again? I did and still get the same error I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 0-glusterd: Received rebalance volume 1 on dccbackup [2011-05-24 04:42:18.521586] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1022) KL > On Tue, May 24, 2011 at 10:02 AM, Karim Latouche > <klatouche at digitalchocolate.com> wrote: >> Hi >> >> Sorry i forgot to include the log message I got when running a >> gluster volume rebalance >> >>> I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] >>> 0-glusterd: Received rebalance volume 1 on dccbackup >>> [2011-05-24 03:59:23.645067] W >>> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading >>> from socket failed. Error (Transport endpoint is not connected), peer >>> (127.0.0.1:1019) >>> [2011-05-24 03:59:28.323938] I >>> [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 0-glusterd: >>> Received rebalance volume 1 on dccbackup >>> [2011-05-24 03:59:28.352874] W >>> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading >>> from socket failed. Error (Transport endpoint is not connected), peer >>> (127.0.0.1:1018) >> And this is the output of the command >> >> gluster volume rebalance dccbackup start >> starting rebalance on volume dccbackup has been unsuccessful >> >> Thx >> >> KL >> >>> Look in the gluster logs and see what it says. /var/log/gluster* >>> >>> If you intention is to migrate data then alternatively you can also do: >>> >>> gluster volume rebalance VOLNAME start >>> >>> >>> On Tue, May 24, 2011 at 2:59 AM, Karim Latouche >>> <klatouche at digitalchocolate.com> wrote: >>>> Hi >>>> >>>> Thx for your help. >>>> >>>> I finaly could add the last brick . >>>> >>>> I now have a Distributed-replicated volume >>>> >>>>> gluster volume info >>>>> >>>>> Volume Name: dccbackup >>>>> Type: Distributed-Replicate >>>>> Status: Started >>>>> Number of Bricks: 2 x 2 = 4 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: mysqldccbackup:/export >>>>> Brick2: mysqldccbackupmirror:/export >>>>> Brick3: gfsbrick1:/brick1 >>>>> Brick4: gfsbrick2:/brick2 >>>> On the client side though I don't see any increasing in storage size. >>>> I tried to rebalance >>>> >>>> gluster volume rebalance dccbackup fix-layout start >>>> >>>> and got . >>>> >>>>> starting rebalance on volume dccbackup has been unsuccessful >>>> I'm kind of stuck here. >>>> >>>> Thank you >>>> >>>> KL >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> What is the output of: >>>>> >>>>> # ls /etc/glusterd/peers >>>>> 39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> 4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> >>>>> Above output is mine. >>>>> >>>>> If you cat each of these files, there should be one for each peer OTHER >>>>> than the host that your are logged into), and hostnames each files, >>>>> like >>>>> this: >>>>> # cat /etc/glusterd/peers/* >>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> state=3 >>>>> hostname1=jc1letgfs15-pfs1 >>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> state=3 >>>>> hostname1=jc1letgfs18-pfs1 >>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> state=3 >>>>> hostname1=jc1letgfs14-pfs1 >>>>> >>>>> Here is a listing of all the peer files on all four nodes of my >>>>> GlusterFS >>>>> servers - I created this configuration in the same way you did - two >>>>> servers, then add two servers: >>>>> >>>>> jc1letgfs14 >>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> state=3 >>>>> hostname1=jc1letgfs15-pfs1 >>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> state=3 >>>>> hostname1=jc1letgfs18-pfs1 >>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>>> state=3 >>>>> hostname1=10.20.72.191 >>>>> >>>>> jc1letgfs15 >>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> state=3 >>>>> hostname1=jc1letgfs18-pfs1 >>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>>> state=3 >>>>> hostname1=10.20.72.191 >>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> state=3 >>>>> hostname1=jc1letgfs14-pfs1 >>>>> >>>>> jc1letgfs17 >>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> state=3 >>>>> hostname1=jc1letgfs15-pfs1 >>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f >>>>> state=3 >>>>> hostname1=jc1letgfs18-pfs1 >>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> state=3 >>>>> hostname1=jc1letgfs14-pfs1 >>>>> >>>>> jc1letgfs18 >>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe >>>>> state=3 >>>>> hostname1=jc1letgfs15-pfs1 >>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156 >>>>> state=3 >>>>> hostname1=10.20.72.191 >>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433 >>>>> state=3 >>>>> hostname1=jc1letgfs14-pfs1 >>>>> >>>>> I hope this is of some help. >>>>> >>>>> James Burnash >>>>> Unix Engineer >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: gluster-users-bounces at gluster.org >>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Karim Latouche >>>>> Sent: Monday, May 23, 2011 1:02 PM >>>>> To: Mohit Anchlia >>>>> Cc: gluster-users at gluster.org >>>>> Subject: Re: Trouble to add brick >>>>> >>>>> hi >>>>> >>>>> Here is the output of gluster peer status command of the four nodes >>>>> >>>>> mysqldccbackup mysqldccbackupmirror are the two first nodes >>>>> >>>>> I got issues when I tried to add gfsbrick1 and two to the already >>>>> working >>>>> cluster >>>>> >>>>> >>>>> >>>>>> [root at gfsbrick1 ~]# gluster peer status >>>>>> Number of Peers: 1 >>>>>> >>>>>> Hostname: 10.0.4.225 >>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>> State: Establishing Connection (Connected) >>>>>> [root at gfsbrick2 ~]# gluster peer status >>>>>> Number of Peers: 2 >>>>>> >>>>>> Hostname: 10.0.4.225 >>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6 >>>>>> State: Peer in Cluster (Connected) >>>>>> >>>>>> Hostname: mysqldccbackupmirror >>>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939 >>>>>> State: Peer Rejected (Connected) >>>>>> [root at mysqldccbackup ~]# gluster peer status >>>>>> Number of Peers: 3 >>>>>> >>>>>> Hostname: mysqldccbackupmirror >>>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939 >>>>>> State: Peer in Cluster (Connected) >>>>>> >>>>>> Hostname: gfsbrick2 >>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f >>>>>> State: Peer in Cluster (Connected) >>>>>> >>>>>> Hostname: gfsbrick1 >>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>> State: Establishing Connection (Connected) >>>>>> root at mysqldccbackupmirror ~]# gluster peer status >>>>>> Number of Peers: 1 >>>>>> >>>>>> Hostname: 10.0.4.225 >>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6 >>>>>> State: Peer in Cluster (Connected) >>>>>> [root at mysqldccbackupmirror ~]# gluster peer status >>>>>> Number of Peers: 1 >>>>>> >>>>>> Hostname: 10.0.4.225 >>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6 >>>>>> State: Peer in Cluster (Connected) >>>>> Thx a lot >>>>> >>>>> KL >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Please send the complete output of gluster peer status from both the >>>>>> nodes. >>>>>> >>>>>> On Mon, May 23, 2011 at 9:44 AM, Karim Latouche >>>>>> <klatouche at digitalchocolate.com> wrote: >>>>>>> hi >>>>>>> >>>>>>> Thx for your answer. >>>>>>> >>>>>>> I should have mention that I already set up a proper hosts file to >>>>>>> avoid >>>>>>> DNS >>>>>>> resolution issue but I still have same issue. >>>>>>> >>>>>>> The last brick is allways the one having issues to be added . I tried >>>>>>> adding >>>>>>> gfsbrick2 first then gfsbrick1 and the result is the same >>>>>>> >>>>>>> I get >>>>>>> >>>>>>>> Hostname: gfsbrick2 >>>>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f >>>>>>>> State: Peer in Cluster (Connected) >>>>>>>> >>>>>>>> Hostname: gfsbrick1 >>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>>>> State: Establishing Connection (Connected) >>>>>>> instead of >>>>>>> >>>>>>> >>>>>>>> Hostname: gfsbrick2 >>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>>>> State: Establishing Connection (Connected) >>>>>>> It 's really strange. >>>>>>> >>>>>>> >>>>>>> Thx >>>>>>> >>>>>>> KL >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Karim, >>>>>>>> gfsbrick1 is not able to contact gfsbrick2 yet. Generally >>>>>>>> happens >>>>>>>> if >>>>>>>> the DNS resolutions dont happen as expected. You can map them >>>>>>>> manually >>>>>>>> in >>>>>>>> /etc/hosts for both the machines?. >>>>>>>> >>>>>>>> something like: >>>>>>>> on gfsbrick1 >>>>>>>> <ip-of-brick2> gfsbrick2 >>>>>>>> on gfsbrick2 >>>>>>>> <ip-of-brick1> gfsbrick1 >>>>>>>> >>>>>>>> Pranith. >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Karim Latouche"<klatouche at digitalchocolate.com> >>>>>>>> To: gluster-users at gluster.org >>>>>>>> Sent: Monday, May 23, 2011 7:56:22 PM >>>>>>>> Subject: Trouble to add brick >>>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> I got a strange problems >>>>>>>> >>>>>>>> I run a two bricks replicated volume and I tried to add 2 more bricks >>>>>>>> >>>>>>>> The first one went ok >>>>>>>> >>>>>>>>> gluster peer probe gfsbrick1 >>>>>>>>> Probe successful >>>>>>>> It seems I can't add the second one >>>>>>>> >>>>>>>> I get a >>>>>>>> >>>>>>>>> Hostname: gfsbrick2 >>>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000 >>>>>>>>> State: Establishing Connection (Connected) >>>>>>>> Did anyone encounter this issue ? >>>>>>>> >>>>>>>> Thx >>>>>>>> >>>>>>>> KL >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>> >>>>> >>>>> DISCLAIMER: >>>>> This e-mail, and any attachments thereto, is intended only for use by >>>>> the >>>>> addressee(s) named herein and may contain legally privileged and/or >>>>> confidential information. If you are not the intended recipient of this >>>>> e-mail, you are hereby notified that any dissemination, distribution or >>>>> copying of this e-mail, and any attachments thereto, is strictly >>>>> prohibited. >>>>> If you have received this in error, please immediately notify me and >>>>> permanently delete the original and any copy of any e-mail and any >>>>> printout >>>>> thereof. E-mail transmission cannot be guaranteed to be secure or >>>>> error-free. The sender therefore does not accept liability for any >>>>> errors or >>>>> omissions in the contents of this message which arise as a result of >>>>> e-mail >>>>> transmission. >>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, >>>>> at >>>>> its discretion, monitor and review the content of all e-mail >>>>> communications. >>>>> http://www.knight.com >>
