Hi abhishek, Not yet, I was busy with some other stuff. Will let you know about it. Thanks, ~Gaurav ----- Original Message ----- From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> To: "Gaurav Garg" <ggarg@xxxxxxxxxx> Cc: gluster-users@xxxxxxxxxxx Sent: Tuesday, March 1, 2016 5:57:12 PM Subject: Re: Issue in Adding/Removing the gluster node Hi Gaurav, Have you got the time to analyze the logs. Regards, Abhishek On Thu, Feb 25, 2016 at 11:23 AM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote: > sure, > > Thanks, > ~Gaurav > > ---- Original Message ----- > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > Cc: gluster-users@xxxxxxxxxxx > Sent: Thursday, February 25, 2016 10:40:11 AM > Subject: Re: Issue in Adding/Removing the gluster node > > Hi Gaurav, > > > Here, I am sharing the log.zip file having logs for both of the nodes and > other logs as well. > > Now I think we can analyze the logs and find out the actual problem of this > issue. > > Regards, > Abhishek > > On Wed, Feb 24, 2016 at 2:44 PM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote: > > > hi abhishek, > > > > i need to look further why are you falling in this situation. file name > > and uuid in /var/lib/glusterd/peers should be same. each file in > > /var/lib/glusterd/peers having information about its peer in the cluster. > > > > could you join #gluster channel on freenode. just ping me (irc name: > > ggarg) after joining the channel. > > > > Thanks, > > Gaurav > > > > > > ----- Original Message ----- > > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > Cc: gluster-users@xxxxxxxxxxx > > Sent: Wednesday, February 24, 2016 12:31:51 PM > > Subject: Re: Issue in Adding/Removing the gluster node > > > > Hi Gaurav, > > > > I have noticed one more thing in etc-glusterfs-glusterd.vol.log file with > > respect to UUID of Peer <10.32.1.144> > > It has two UUID > > Before removing > > > > UUID is - b88c74b9-457d-4864-9fe6-403f6934d7d1 and after inserting the > node > > UUID is - 5ec06937-5f85-4a9d-b29e-4227bbb7b4fa > > > > Also have one file in glusterd/peers/ directory with the same name of > first > > UUID. > > > > What does this file mean in peers directory? is this file providing some > > kind of linking between both of the UUID? > > > > Please find this file as an attachment. > > > > Regards, > > Abhishek > > > > On Wed, Feb 24, 2016 at 12:06 PM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote: > > > > > Hi abhishek, > > > > > > yes i looked into configuration file's that you have provided. there > > every > > > things seems to be fine. > > > > > > seems like some other problem. i will look into it today and will come > > > back to you. > > > > > > thanks, > > > > > > ~Gaurav > > > > > > ----- Original Message ----- > > > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > Cc: gluster-users@xxxxxxxxxxx > > > Sent: Wednesday, February 24, 2016 12:02:47 PM > > > Subject: Re: Issue in Adding/Removing the gluster node > > > > > > Hi Gaurav, > > > > > > Have you get the time to see the logs files which you asked yesterday? > > > > > > Regards, > > > Abhishek > > > > > > On Tue, Feb 23, 2016 at 3:05 PM, ABHISHEK PALIWAL < > > abhishpaliwal@xxxxxxxxx > > > > > > > wrote: > > > > > > > Hi Gaurav, > > > > > > > > Please find the vol.tar file. > > > > > > > > Regards, > > > > Abhishek > > > > > > > > On Tue, Feb 23, 2016 at 2:37 PM, Gaurav Garg <ggarg@xxxxxxxxxx> > wrote: > > > > > > > >> Hi abhishek, > > > >> > > > >> >> But after analyzing the following logs from the 1st board seems > > that > > > >> the > > > >> process which will update the second brick in output of "# gluster > > > volume > > > >> status c_glusterfs" takes sometime to update this table and before > the > > > >> updation of this table remove-brick is getting executed that is why > it > > > is > > > >> getting failed. > > > >> > > > >> It should not take that much of time. If your peer probe is > successful > > > >> and you are able to > > > >> see 2nd broad peer entry in #gluster peer status command then it > have > > > >> updated all information > > > >> of volume internally. > > > >> > > > >> your gluster volume status showing 2nd board entry: > > > >> > > > >> Brick 10.32.0.48:/opt/lvmdir/c2/brick 49153 0 Y > > > >> 2537 > > > >> Self-heal Daemon on localhost N/A N/A Y > > > >> 5577 > > > >> Self-heal Daemon on 10.32.1.144 N/A N/A Y > > > >> 3850 > > > >> > > > >> but its not showing 2nd board brick entry. > > > >> > > > >> > > > >> Did you perform any manual operation with configuration file which > > > >> resides in /var/lib/glusterd/* ? > > > >> > > > >> could you attach/paste the file > > > >> /var/lib/glusterd/vols/c_glusterfs/trusted-*.tcp-fuse.vol file. > > > >> > > > >> > > > >> Thanks, > > > >> > > > >> Regards, > > > >> Gaurav > > > >> > > > >> ----- Original Message ----- > > > >> From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> Cc: gluster-users@xxxxxxxxxxx > > > >> Sent: Tuesday, February 23, 2016 1:33:30 PM > > > >> Subject: Re: Issue in Adding/Removing the gluster > node > > > >> > > > >> Hi Gaurav, > > > >> > > > >> For the network connectivity I am doing peer probe to the > 10.32.1.144 > > > i.e. > > > >> 2nd board thats working fine means connectivity is there. > > > >> > > > >> #peer probe 10.32.1.144 > > > >> > > > >> if the above command get success > > > >> > > > >> I executed the the remove-brick command which is getting failed. > > > >> > > > >> So, now it seems the the peer probe will not give the correct > > > >> connectivity > > > >> status to execute the remove-brick command. > > > >> > > > >> But after analyzing the following logs from the 1st board seems that > > the > > > >> process which will update the second brick in output of "# gluster > > > volume > > > >> status c_glusterfs" takes sometime to update this table and before > the > > > >> updation of this table remove-brick is getting executed that is why > it > > > is > > > >> getting failed. > > > >> > > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > >> > > > >> *1st board:* > > > >> # gluster volume info > > > >> status > > > >> gluster volume status c_glusterfs > > > >> Volume Name: c_glusterfs > > > >> Type: Replicate > > > >> Volume ID: 32793e91-6f88-4f29-b3e4-0d53d02a4b99 > > > >> Status: Started > > > >> Number of Bricks: 1 x 2 = 2 > > > >> Transport-type: tcp > > > >> Bricks: > > > >> Brick1: 10.32.0.48:/opt/lvmdir/c2/brick > > > >> Brick2: 10.32.1.144:/opt/lvmdir/c2/brick > > > >> Options Reconfigured: > > > >> nfs.disable: on > > > >> network.ping-timeout: 4 > > > >> performance.readdir-ahead: on > > > >> # gluster peer status > > > >> Number of Peers: 1 > > > >> > > > >> Hostname: 10.32.1.144 > > > >> Uuid: b88c74b9-457d-4864-9fe6-403f6934d7d1 > > > >> State: Peer in Cluster (Connected) > > > >> # gluster volume status c_glusterfs > > > >> Status of volume: c_glusterfs > > > >> Gluster process TCP Port RDMA Port > > Online > > > >> Pid > > > >> > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> > > > >> Brick 10.32.0.48:/opt/lvmdir/c2/brick 49153 0 Y > > > >> 2537 > > > >> Self-heal Daemon on localhost N/A N/A Y > > > >> 5577 > > > >> Self-heal Daemon on 10.32.1.144 N/A N/A Y > > > >> 3850 > > > >> > > > >> Task Status of Volume c_glusterfs > > > >> > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> > > > >> There are no active volume tasks > > > >> > > > >> +++++++++++++++++++++++++++++++++++++++++++++++ > > > >> > > > >> I'll try this with some delay or wait to remove-brick until the # > > > gluster > > > >> volume status c_glusterfs command show second brick in the list. > > > >> > > > >> May we this approach will resolve the issue. > > > >> > > > >> Please comment, If you are agree with my observation > > > >> > > > >> Regards, > > > >> Abhishek > > > >> > > > >> On Tue, Feb 23, 2016 at 1:10 PM, ABHISHEK PALIWAL < > > > >> abhishpaliwal@xxxxxxxxx> > > > >> wrote: > > > >> > > > >> > Hi Gaurav, > > > >> > > > > >> > In my case we are removing the brick in the offline state with the > > > force > > > >> > option like in the following way: > > > >> > > > > >> > > > > >> > > > > >> > *gluster volume remove-brick %s replica 1 %s:%s force > --mode=script* > > > >> > but still getting the failure or remove-brick > > > >> > > > > >> > it seems that brick is not present which we are trying to remove > > here > > > >> are > > > >> > the log snippet of both of the boards > > > >> > > > > >> > > > > >> > *1st board:* > > > >> > # gluster volume info > > > >> > status > > > >> > gluster volume status c_glusterfs > > > >> > Volume Name: c_glusterfs > > > >> > Type: Replicate > > > >> > Volume ID: 32793e91-6f88-4f29-b3e4-0d53d02a4b99 > > > >> > Status: Started > > > >> > Number of Bricks: 1 x 2 = 2 > > > >> > Transport-type: tcp > > > >> > Bricks: > > > >> > Brick1: 10.32.0.48:/opt/lvmdir/c2/brick > > > >> > Brick2: 10.32.1.144:/opt/lvmdir/c2/brick > > > >> > Options Reconfigured: > > > >> > nfs.disable: on > > > >> > network.ping-timeout: 4 > > > >> > performance.readdir-ahead: on > > > >> > # gluster peer status > > > >> > Number of Peers: 1 > > > >> > > > > >> > Hostname: 10.32.1.144 > > > >> > Uuid: b88c74b9-457d-4864-9fe6-403f6934d7d1 > > > >> > State: Peer in Cluster (Connected) > > > >> > # gluster volume status c_glusterfs > > > >> > Status of volume: c_glusterfs > > > >> > Gluster process TCP Port RDMA Port > > > Online > > > >> > Pid > > > >> > > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> > > > > >> > Brick 10.32.0.48:/opt/lvmdir/c2/brick 49153 0 > Y > > > >> > 2537 > > > >> > Self-heal Daemon on localhost N/A N/A Y > > > >> > 5577 > > > >> > Self-heal Daemon on 10.32.1.144 N/A N/A Y > > > >> > 3850 > > > >> > > > > >> > Task Status of Volume c_glusterfs > > > >> > > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> > > > > >> > There are no active volume tasks > > > >> > > > > >> > *2nd Board*: > > > >> > > > > >> > # gluster volume info > > > >> > status > > > >> > gluster volume status c_glusterfs > > > >> > gluster volume heal c_glusterfs info > > > >> > > > > >> > Volume Name: c_glusterfs > > > >> > Type: Replicate > > > >> > Volume ID: 32793e91-6f88-4f29-b3e4-0d53d02a4b99 > > > >> > Status: Started > > > >> > Number of Bricks: 1 x 2 = 2 > > > >> > Transport-type: tcp > > > >> > Bricks: > > > >> > Brick1: 10.32.0.48:/opt/lvmdir/c2/brick > > > >> > Brick2: 10.32.1.144:/opt/lvmdir/c2/brick > > > >> > Options Reconfigured: > > > >> > performance.readdir-ahead: on > > > >> > network.ping-timeout: 4 > > > >> > nfs.disable: on > > > >> > # gluster peer status > > > >> > Number of Peers: 1 > > > >> > > > > >> > Hostname: 10.32.0.48 > > > >> > Uuid: e7c4494e-aa04-4909-81c9-27a462f6f9e7 > > > >> > State: Peer in Cluster (Connected) > > > >> > # gluster volume status c_glusterfs > > > >> > Status of volume: c_glusterfs > > > >> > Gluster process TCP Port RDMA Port > > > Online > > > >> > Pid > > > >> > > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> > > > > >> > Brick 10.32.0.48:/opt/lvmdir/c2/brick 49153 0 > Y > > > >> > 2537 > > > >> > Self-heal Daemon on localhost N/A N/A Y > > > >> > 3850 > > > >> > Self-heal Daemon on 10.32.0.48 N/A N/A Y > > > >> > 5577 > > > >> > > > > >> > Task Status of Volume c_glusterfs > > > >> > > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> > > > > >> > There are no active volume tasks > > > >> > > > > >> > Do you know why these logs are not showing the Brick info at the > > time > > > of > > > >> > gluster volume status. > > > >> > Because we are not able to collect the logs of cmd_history.log > file > > > from > > > >> > the 2nd board. > > > >> > > > > >> > Regards, > > > >> > Abhishek > > > >> > > > > >> > > > > >> > On Tue, Feb 23, 2016 at 12:02 PM, Gaurav Garg <ggarg@xxxxxxxxxx> > > > wrote: > > > >> > > > > >> >> Hi abhishek, > > > >> >> > > > >> >> >> Can we perform remove-brick operation on the offline brick? > what > > > is > > > >> the > > > >> >> meaning of offline and online brick? > > > >> >> > > > >> >> No, you can't perform remove-brick operation on the offline > brick. > > > >> brick > > > >> >> is offline means brick process is not running. you can see it by > > > >> executing > > > >> >> #gluster volume status. If brick is offline then respective brick > > > will > > > >> show > > > >> >> "N" entry in Online column of #gluster volume status command. > > > >> Alternatively > > > >> >> you can also check whether glusterfsd process for that brick is > > > >> running or > > > >> >> not by executing #ps aux | grep glusterfsd, this command will > list > > > out > > > >> all > > > >> >> the brick process you can filter out from them, which one is > > online, > > > >> which > > > >> >> one is not. > > > >> >> > > > >> >> But if you want to perform remove-brick operation on the offline > > > brick > > > >> >> then you need to execute it with force option. #gluster volume > > > >> remove-brick > > > >> >> <volname> hostname:/brick_name force. This might lead to data > loss. > > > >> >> > > > >> >> > > > >> >> > > > >> >> >> Also, Is there any logic in gluster through which we can check > > the > > > >> >> connectivity of node established or not before performing the any > > > >> >> operation > > > >> >> on brick? > > > >> >> > > > >> >> Yes, you can check it by executing #gluster peer status command. > > > >> >> > > > >> >> > > > >> >> Thanks, > > > >> >> > > > >> >> ~Gaurav > > > >> >> > > > >> >> > > > >> >> ----- Original Message ----- > > > >> >> From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> >> Cc: gluster-users@xxxxxxxxxxx > > > >> >> Sent: Tuesday, February 23, 2016 11:50:43 AM > > > >> >> Subject: Re: Issue in Adding/Removing the gluster > > > node > > > >> >> > > > >> >> Hi Gaurav, > > > >> >> > > > >> >> one general question related to gluster bricks. > > > >> >> > > > >> >> Can we perform remove-brick operation on the offline brick? what > is > > > the > > > >> >> meaning of offline and online brick? > > > >> >> Also, Is there any logic in gluster through which we can check > the > > > >> >> connectivity of node established or not before performing the any > > > >> >> operation > > > >> >> on brick? > > > >> >> > > > >> >> Regards, > > > >> >> Abhishek > > > >> >> > > > >> >> On Mon, Feb 22, 2016 at 2:42 PM, Gaurav Garg <ggarg@xxxxxxxxxx> > > > wrote: > > > >> >> > > > >> >> > Hi abhishek, > > > >> >> > > > > >> >> > I went through your logs of node 1 and by looking glusterd logs > > its > > > >> >> > clearly indicate that your 2nd node (10.32.1.144) have > > disconnected > > > >> from > > > >> >> > the cluster, because of that remove-brick operation failed. I > > think > > > >> you > > > >> >> > need to check your network interface. > > > >> >> > > > > >> >> > But surprising things is that i did not see duplicate peer > entry > > in > > > >> >> > #gluster peer status command output. > > > >> >> > > > > >> >> > May be i will get some more information from your (10.32.1.144) > > 2nd > > > >> node > > > >> >> > logs. Could you also attach your 2nd node logs. > > > >> >> > > > > >> >> > after restarting glusterd, are you seeing duplicate peer entry > in > > > >> >> #gluster > > > >> >> > peer status command output ? > > > >> >> > > > > >> >> > will wait for 2nd node logs for further analyzing duplicate > peer > > > >> entry > > > >> >> > problem. > > > >> >> > > > > >> >> > Thanks, > > > >> >> > > > > >> >> > ~Gaurav > > > >> >> > > > > >> >> > ----- Original Message ----- > > > >> >> > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> > To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> >> > Cc: gluster-users@xxxxxxxxxxx > > > >> >> > Sent: Monday, February 22, 2016 12:48:55 PM > > > >> >> > Subject: Re: Issue in Adding/Removing the > gluster > > > >> node > > > >> >> > > > > >> >> > Hi Gaurav, > > > >> >> > > > > >> >> > Here, You can find the attached logs for the boards in case of > > > >> >> remove-brick > > > >> >> > failure. > > > >> >> > In these logs we do not have the cmd_history and > > > >> >> > etc-glusterfs-glusterd.vol.log for the second board. > > > >> >> > > > > >> >> > May be for that we need to some more time. > > > >> >> > > > > >> >> > > > > >> >> > Regards, > > > >> >> > Abhishek > > > >> >> > > > > >> >> > On Mon, Feb 22, 2016 at 10:18 AM, Gaurav Garg < > ggarg@xxxxxxxxxx> > > > >> wrote: > > > >> >> > > > > >> >> > > Hi Abhishek, > > > >> >> > > > > > >> >> > > >> I'll provide the required log to you. > > > >> >> > > > > > >> >> > > sure > > > >> >> > > > > > >> >> > > on both node. do "pkill glusterd" and then start glusterd > > > services. > > > >> >> > > > > > >> >> > > Thanks, > > > >> >> > > > > > >> >> > > ~Gaurav > > > >> >> > > > > > >> >> > > ----- Original Message ----- > > > >> >> > > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> > > To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> >> > > Cc: gluster-users@xxxxxxxxxxx > > > >> >> > > Sent: Monday, February 22, 2016 10:11:48 AM > > > >> >> > > Subject: Re: Issue in Adding/Removing the > > gluster > > > >> node > > > >> >> > > > > > >> >> > > Hi Gaurav, > > > >> >> > > > > > >> >> > > Thanks for your prompt reply. > > > >> >> > > > > > >> >> > > I'll provide the required log to you. > > > >> >> > > > > > >> >> > > As a workaround you suggested that restart the glusterd > > service. > > > >> Could > > > >> >> > you > > > >> >> > > please tell me the point where I can do this? > > > >> >> > > > > > >> >> > > Regards, > > > >> >> > > Abhishek > > > >> >> > > > > > >> >> > > On Fri, Feb 19, 2016 at 6:11 PM, Gaurav Garg < > ggarg@xxxxxxxxxx > > > > > > >> >> wrote: > > > >> >> > > > > > >> >> > > > Hi Abhishek, > > > >> >> > > > > > > >> >> > > > Peer status output looks interesting where it have stale > > entry, > > > >> >> > > > technically it should not happen. Here few thing need to > ask > > > >> >> > > > > > > >> >> > > > Did you perform any manual operation with GlusterFS > > > configuration > > > >> >> file > > > >> >> > > > which resides in /var/lib/glusterd/* folder. > > > >> >> > > > > > > >> >> > > > Can you provide output of "ls /var/lib/glusterd/peers" > from > > > >> both of > > > >> >> > your > > > >> >> > > > nodes. > > > >> >> > > > > > > >> >> > > > Could you provide output of #gluster peer status command > when > > > 2nd > > > >> >> node > > > >> >> > is > > > >> >> > > > down > > > >> >> > > > > > > >> >> > > > Can you provide output of #gluster volume info command > > > >> >> > > > > > > >> >> > > > Can you provide full logs details of cmd_history.log and > > > >> >> > > > etc-glusterfs-glusterd.vol.log from both the nodes. > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > You can restart your glusterd as of now as a workaround but > > we > > > >> need > > > >> >> to > > > >> >> > > > analysis this issue further. > > > >> >> > > > > > > >> >> > > > Thanks, > > > >> >> > > > Gaurav > > > >> >> > > > > > > >> >> > > > ----- Original Message ----- > > > >> >> > > > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> > > > To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> >> > > > Cc: gluster-users@xxxxxxxxxxx > > > >> >> > > > Sent: Friday, February 19, 2016 5:27:21 PM > > > >> >> > > > Subject: Re: Issue in Adding/Removing the > > > gluster > > > >> >> node > > > >> >> > > > > > > >> >> > > > Hi Gaurav, > > > >> >> > > > > > > >> >> > > > After the failure of add-brick following is outcome > "gluster > > > peer > > > >> >> > status" > > > >> >> > > > command > > > >> >> > > > > > > >> >> > > > Number of Peers: 2 > > > >> >> > > > > > > >> >> > > > Hostname: 10.32.1.144 > > > >> >> > > > Uuid: bbe2a458-ad3d-406d-b233-b6027c12174e > > > >> >> > > > State: Peer in Cluster (Connected) > > > >> >> > > > > > > >> >> > > > Hostname: 10.32.1.144 > > > >> >> > > > Uuid: bbe2a458-ad3d-406d-b233-b6027c12174e > > > >> >> > > > State: Peer in Cluster (Connected) > > > >> >> > > > > > > >> >> > > > Regards, > > > >> >> > > > Abhishek > > > >> >> > > > > > > >> >> > > > On Fri, Feb 19, 2016 at 5:21 PM, ABHISHEK PALIWAL < > > > >> >> > > abhishpaliwal@xxxxxxxxx > > > >> >> > > > > > > > >> >> > > > wrote: > > > >> >> > > > > > > >> >> > > > > Hi Gaurav, > > > >> >> > > > > > > > >> >> > > > > Both are the board connect through the backplane using > > > >> ethernet. > > > >> >> > > > > > > > >> >> > > > > Even this inconsistency also occurs when I am trying to > > > >> bringing > > > >> >> back > > > >> >> > > the > > > >> >> > > > > node in slot. Means some time add-brick executes without > > > >> failure > > > >> >> but > > > >> >> > > some > > > >> >> > > > > time following error occurs. > > > >> >> > > > > > > > >> >> > > > > volume add-brick c_glusterfs replica 2 10.32.1.144: > > > >> >> > > /opt/lvmdir/c2/brick > > > >> >> > > > > force : FAILED : Another transaction is in progress for > > > >> >> c_glusterfs. > > > >> >> > > > Please > > > >> >> > > > > try again after sometime. > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > You can also see the attached logs for add-brick failure > > > >> scenario. > > > >> >> > > > > > > > >> >> > > > > Please let me know if you need more logs. > > > >> >> > > > > > > > >> >> > > > > Regards, > > > >> >> > > > > Abhishek > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > On Fri, Feb 19, 2016 at 5:03 PM, Gaurav Garg < > > > ggarg@xxxxxxxxxx > > > >> > > > > >> >> > wrote: > > > >> >> > > > > > > > >> >> > > > >> Hi Abhishek, > > > >> >> > > > >> > > > >> >> > > > >> How are you connecting two board, and how are you > removing > > > it > > > >> >> > manually > > > >> >> > > > >> that need to know because if you are removing your 2nd > > board > > > >> from > > > >> >> > the > > > >> >> > > > >> cluster (abrupt shutdown) then you can't perform remove > > > brick > > > >> >> > > operation > > > >> >> > > > in > > > >> >> > > > >> 2nd node from first node and its happening successfully > in > > > >> your > > > >> >> > case. > > > >> >> > > > could > > > >> >> > > > >> you ensure your network connection once again while > > removing > > > >> and > > > >> >> > > > bringing > > > >> >> > > > >> back your node again. > > > >> >> > > > >> > > > >> >> > > > >> Thanks, > > > >> >> > > > >> Gaurav > > > >> >> > > > >> > > > >> >> > > > >> ------------------------------ > > > >> >> > > > >> *From: *"ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> > > > >> *To: *"Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> >> > > > >> *Cc: *gluster-users@xxxxxxxxxxx > > > >> >> > > > >> *Sent: *Friday, February 19, 2016 3:36:21 PM > > > >> >> > > > >> > > > >> >> > > > >> *Subject: *Re: Issue in Adding/Removing > > the > > > >> >> gluster > > > >> >> > > node > > > >> >> > > > >> > > > >> >> > > > >> Hi Gaurav, > > > >> >> > > > >> > > > >> >> > > > >> Thanks for reply > > > >> >> > > > >> > > > >> >> > > > >> 1. Here, I removed the board manually here but this time > > it > > > >> works > > > >> >> > fine > > > >> >> > > > >> > > > >> >> > > > >> [2016-02-18 10:03:40.601472] : volume remove-brick > > > >> c_glusterfs > > > >> >> > > replica > > > >> >> > > > 1 > > > >> >> > > > >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > > > >> >> > > > >> [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 > : > > > >> SUCCESS > > > >> >> > > > >> > > > >> >> > > > >> Yes this time board is reachable but how? don't know > > because > > > >> >> board > > > >> >> > is > > > >> >> > > > >> detached. > > > >> >> > > > >> > > > >> >> > > > >> 2. Here, I attached the board this time its works fine > in > > > >> >> add-bricks > > > >> >> > > > >> > > > >> >> > > > >> 2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : > > > >> SUCCESS > > > >> >> > > > >> [2016-02-18 10:03:44.563546] : volume add-brick > > c_glusterfs > > > >> >> > replica 2 > > > >> >> > > > >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > > > >> >> > > > >> > > > >> >> > > > >> 3.Here, again I removed the board this time failed occur > > > >> >> > > > >> > > > >> >> > > > >> [2016-02-18 10:37:02.816089] : volume remove-brick > > > >> c_glusterfs > > > >> >> > > replica > > > >> >> > > > 1 > > > >> >> > > > >> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : > > Incorrect > > > >> >> brick > > > >> >> > > > >> 10.32.1.144:/opt > > > >> >> > > > >> /lvmdir/c2/brick for volume c_glusterfs > > > >> >> > > > >> > > > >> >> > > > >> but here board is not reachable. > > > >> >> > > > >> > > > >> >> > > > >> why this inconsistency is there while doing the same > step > > > >> >> multiple > > > >> >> > > time. > > > >> >> > > > >> > > > >> >> > > > >> Hope you are getting my point. > > > >> >> > > > >> > > > >> >> > > > >> Regards, > > > >> >> > > > >> Abhishek > > > >> >> > > > >> > > > >> >> > > > >> On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg < > > > >> ggarg@xxxxxxxxxx> > > > >> >> > > wrote: > > > >> >> > > > >> > > > >> >> > > > >>> Abhishek, > > > >> >> > > > >>> > > > >> >> > > > >>> when sometime its working fine means 2nd board network > > > >> >> connection > > > >> >> > is > > > >> >> > > > >>> reachable to first node. you can conform this by > > executing > > > >> same > > > >> >> > > > #gluster > > > >> >> > > > >>> peer status command. > > > >> >> > > > >>> > > > >> >> > > > >>> Thanks, > > > >> >> > > > >>> Gaurav > > > >> >> > > > >>> > > > >> >> > > > >>> ----- Original Message ----- > > > >> >> > > > >>> From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> > > > >>> To: "Gaurav Garg" <ggarg@xxxxxxxxxx> > > > >> >> > > > >>> Cc: gluster-users@xxxxxxxxxxx > > > >> >> > > > >>> Sent: Friday, February 19, 2016 3:12:22 PM > > > >> >> > > > >>> Subject: Re: Issue in Adding/Removing > the > > > >> >> gluster > > > >> >> > > node > > > >> >> > > > >>> > > > >> >> > > > >>> Hi Gaurav, > > > >> >> > > > >>> > > > >> >> > > > >>> Yes, you are right actually I am force fully detaching > > the > > > >> node > > > >> >> > from > > > >> >> > > > the > > > >> >> > > > >>> slave and when we removed the board it disconnected > from > > > the > > > >> >> > another > > > >> >> > > > >>> board. > > > >> >> > > > >>> > > > >> >> > > > >>> but my question is I am doing this process multiple > time > > > some > > > >> >> time > > > >> >> > it > > > >> >> > > > >>> works > > > >> >> > > > >>> fine but some time it gave these errors. > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> you can see the following logs from cmd_history.log > file > > > >> >> > > > >>> > > > >> >> > > > >>> [2016-02-18 10:03:34.497996] : volume set c_glusterfs > > > >> >> nfs.disable > > > >> >> > > on : > > > >> >> > > > >>> SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:34.915036] : volume start > c_glusterfs > > > >> force : > > > >> >> > > > SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:40.250326] : volume status : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:40.273275] : volume status : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:40.601472] : volume remove-brick > > > >> c_glusterfs > > > >> >> > > > replica 1 > > > >> >> > > > >>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:40.885973] : peer detach > 10.32.1.144 : > > > >> >> SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 > : > > > >> SUCCESS > > > >> >> > > > >>> [2016-02-18 10:03:44.563546] : volume add-brick > > > c_glusterfs > > > >> >> > replica > > > >> >> > > 2 > > > >> >> > > > >>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:30:53.297415] : volume status : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:30:53.313096] : volume status : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:37:02.748714] : volume status : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:37:02.762091] : volume status : SUCCESS > > > >> >> > > > >>> [2016-02-18 10:37:02.816089] : volume remove-brick > > > >> c_glusterfs > > > >> >> > > > replica 1 > > > >> >> > > > >>> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : > > > Incorrect > > > >> >> brick > > > >> >> > > > >>> 10.32.1.144:/opt/lvmdir/c2/brick for volume > c_glusterfs > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg < > > > >> ggarg@xxxxxxxxxx> > > > >> >> > > wrote: > > > >> >> > > > >>> > > > >> >> > > > >>> > Hi Abhishek, > > > >> >> > > > >>> > > > > >> >> > > > >>> > Seems your peer 10.32.1.144 have disconnected while > > doing > > > >> >> remove > > > >> >> > > > brick. > > > >> >> > > > >>> > see the below logs in glusterd: > > > >> >> > > > >>> > > > > >> >> > > > >>> > [2016-02-18 10:37:02.816009] E [MSGID: 106256] > > > >> >> > > > >>> > > > > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] > > > >> >> > > > >>> 0-management: > > > >> >> > > > >>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for > > > >> volume > > > >> >> > > > >>> c_glusterfs > > > >> >> > > > >>> > [Invalid argument] > > > >> >> > > > >>> > [2016-02-18 10:37:02.816061] E [MSGID: 106265] > > > >> >> > > > >>> > > > > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] > > > >> >> > > > >>> 0-management: > > > >> >> > > > >>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for > > > >> volume > > > >> >> > > > >>> c_glusterfs > > > >> >> > > > >>> > The message "I [MSGID: 106004] > > > >> >> > > > >>> > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] > > > >> >> > 0-management: > > > >> >> > > > Peer > > > >> >> > > > >>> > <10.32.1.144> > (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), > > > in > > > >> >> state > > > >> >> > > > <Peer > > > >> >> > > > >>> in > > > >> >> > > > >>> > Cluster>, has disconnected from glusterd." repeated > 25 > > > >> times > > > >> >> > > between > > > >> >> > > > >>> > [2016-02-18 10:35:43.131945] and [2016-02-18 > > > >> 10:36:58.160458] > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > If you are facing the same issue now, could you paste > > > your > > > >> # > > > >> >> > > gluster > > > >> >> > > > >>> peer > > > >> >> > > > >>> > status command output here. > > > >> >> > > > >>> > > > > >> >> > > > >>> > Thanks, > > > >> >> > > > >>> > ~Gaurav > > > >> >> > > > >>> > > > > >> >> > > > >>> > ----- Original Message ----- > > > >> >> > > > >>> > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > > > >> >> > > > >>> > To: gluster-users@xxxxxxxxxxx > > > >> >> > > > >>> > Sent: Friday, February 19, 2016 2:46:35 PM > > > >> >> > > > >>> > Subject: Issue in Adding/Removing the > > > >> gluster > > > >> >> > node > > > >> >> > > > >>> > > > > >> >> > > > >>> > Hi, > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > I am working on two board setup connecting to each > > other. > > > >> >> Gluster > > > >> >> > > > >>> version > > > >> >> > > > >>> > 3.7.6 is running and added two bricks in replica 2 > mode > > > but > > > >> >> when > > > >> >> > I > > > >> >> > > > >>> manually > > > >> >> > > > >>> > removed (detach) the one board from the setup I am > > > getting > > > >> the > > > >> >> > > > >>> following > > > >> >> > > > >>> > error. > > > >> >> > > > >>> > > > > >> >> > > > >>> > volume remove-brick c_glusterfs replica 1 > 10.32.1.144: > > > >> >> > > > >>> /opt/lvmdir/c2/brick > > > >> >> > > > >>> > force : FAILED : Incorrect brick 10.32.1.144: > > > >> >> > /opt/lvmdir/c2/brick > > > >> >> > > > for > > > >> >> > > > >>> > volume c_glusterfs > > > >> >> > > > >>> > > > > >> >> > > > >>> > Please find the logs file as an attachment. > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > Regards, > > > >> >> > > > >>> > Abhishek > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > _______________________________________________ > > > >> >> > > > >>> > Gluster-users mailing list > > > >> >> > > > >>> > Gluster-users@xxxxxxxxxxx > > > >> >> > > > >>> > > http://www.gluster.org/mailman/listinfo/gluster-users > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> -- > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> Regards > > > >> >> > > > >>> Abhishek Paliwal > > > >> >> > > > >>> > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > >> -- > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > >> Regards > > > >> >> > > > >> Abhishek Paliwal > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > -- > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > Regards > > > >> >> > > > Abhishek Paliwal > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > -- > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > Regards > > > >> >> > Abhishek Paliwal > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> -- > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> Regards > > > >> >> Abhishek Paliwal > > > >> >> > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > Regards > > > >> > Abhishek Paliwal > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> > > > >> > > > >> > > > >> > > > >> Regards > > > >> Abhishek Paliwal > > > >> > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Regards > > > > Abhishek Paliwal > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Regards > > > Abhishek Paliwal > > > > > > > > > > > -- > > > > > > > > > > Regards > > Abhishek Paliwal > > > > > > -- > > > > > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
