Re: Clarification about remove-brick

[Mohammed Rafi K C <rkavunga@xxxxxxxxxx>]

On 07/29/2016 11:08 PM, Richard Klein (RSI) wrote:

Thank you very much for the information.  I have read the documentation link you have listed below and just wanted confirmation about the remove-brick process.  I did not see any documentation about the ability to use the remove-brick stop command so that is good to know.

I'm not sure, how much testing has been done in that area. But we do have the remove stop command, and theoretically it should work. If you do rebalance after remove-brick stop, then again the cluster will be load balanced if files are already migrated as part of the remove brick operation.

 

I have a couple of related follow up questions as well:

 

1)      Let’s say a brick fails (hard drive goes bad).  I want to replace the bad drive with a new one and then get it back operational in the cluster.  In this case I don’t have spare brick available so I can’t do the replace-brick.  The only procedure I have found is a blog and bug report for 3.4 about this issue with a work around. Here is the link to the blog: https://joejulian.name/blog/replacing-a-brick-on-glusterfs-340/.  Here is the link to the bug report: https://bugzilla.redhat.com/show_bug.cgi?id=991084.  My apologies if this has been addressed before but I have searched and can’t find a solution where you just replace the bad drive with a good one and recover.  Any instructions about this process would be appreciated.

If I understand you correctly, you want to replace a drive and create the brick with same name. If that is the case, patch http://review.gluster.org/#/c/12250/ will solve the. In fact the patch is under review. CCing the replace brick experts to give you more info regarding your replace queries.

2)      Does the rebalance process lock data files in Gluster?  We are using Gluster as primary storage in Cloudstack 4.7. If we shrink or expand the Gluster volume we do a rebalance of the layout and data.  However, after this process we have several VMs which have disk volumes in a read only state like there were disk problems.  Once we power cycle the VM all is well and no data loss occurred but there seems to be a correlation between the rebalance and the errors.  I am wondering if the rebalance process locks the data somehow and makes it unavailable to the VM.

Rebalance process will take locks as part of the healing. But I'm not sure whether it cause VM's to go into read-only state. The locks will be took  on directory for healing and for files during migration, but only for a specific offset. So both should not cause an entire file to be read-only for a long time.

Can you describe your problem more ? I will try to see if there is anything else causing the issue.


Regards
Rafi KC

 

Thanks again for the response.

 

Richard Klein

RSI

 

 

From: Mohammed Rafi K C [mailto:rkavunga@xxxxxxxxxx]
Sent: Friday, July 29, 2016 1:38 AM
To: Lenovo Lastname; Richard Klein (RSI); gluster-users@xxxxxxxxxxx
Subject: Re: Clarification about remove-brick

 

You can also see the documentation here https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes

 

Rafi KC

On 07/29/2016 11:39 AM, Mohammed Rafi K C wrote:

 

I will summarize the procedure for removing a brick with description.

 

1) start an add brick operation using gluster volume remov-brick command. This command will mark the mentioned brick as a decommissioned brick. Also, this will kick a process that will start migrating data from the decommissioned brick to the other bricks.

2) Once the migration is finished you can safely do a remove-brick commit.

3) Or if you wish to stop the process and reset the decommissioned brick, you can do remove-brick stop. This will not migrate the data back to the decommissioned brick. It will stay in the other bricks and the data will be still accessible, if you want to have proper load balancing after this, you can start rebalance process.

4) If you wish to do an instant remove brick you can use force option, which will not migrate data, hence your whole data in the removed brick will be lost from mount point.

On 07/29/2016 01:25 AM, Lenovo Lastname wrote:

I'm using 3.7.11, this command works with me,

 

!remove-brick
[root@node2 ~]# gluster volume remove-brick v1 replica 2 192.168.3.73:/gfs/b1/v1 force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success

 

Don't know about the commit thingy...

 

On Thursday, July 28, 2016 3:47 PM, Richard Klein (RSI) <rklein@xxxxxxxxxx> wrote:

 

We are using Gluster 3.7.6 in a replica 2 distributed-replicate configuration.  I am wondering when we do a remove-brick with just one brick pair will the data be moved off the bricks once the status show complete and then you do the commit?    Also, if you start a remove-brick process can you stop it?  Is there an abort or stop command or do you just don’t do the commit?

 

Any help would be appreciated.

 

Richard Klein

RSI

 

 

 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

http://www.gluster.org/mailman/listinfo/gluster-users

 

 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users