Re: Disk failed, how do I remove brick?

Nithya Balachandran <nbalacha@xxxxxxxxxx> · Wed, 15 Jun 2016 10:19:45 +0530

On Fri, Jun 10, 2016 at 1:25 AM, Phil Dumont <phil@xxxxxxxxxxxxxxxxxxxxxxxx> wrote:
Just started trying gluster, to decide if we want to put it into production.

Running version 3.7.11-1
Replicated, distributed volume, two servers, 20 bricks per server:

[root@storinator1 ~]# gluster volume status gv0
Status of volume: gv0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick storinator1:/export/brick1/gv0        49153     0          Y       2554 
Brick storinator2:/export/brick1/gv0        49153     0          Y       9686 
Brick storinator1:/export/brick2/gv0        49154     0          Y       2562 
Brick storinator2:/export/brick2/gv0        49154     0          Y       9708 
Brick storinator1:/export/brick3/gv0        49155     0          Y       2568 
Brick storinator2:/export/brick3/gv0        49155     0          Y       9692 
Brick storinator1:/export/brick4/gv0        49156     0          Y       2574 
Brick storinator2:/export/brick4/gv0        49156     0          Y       9765 
Brick storinator1:/export/brick5/gv0        49173     0          Y       16901
Brick storinator2:/export/brick5/gv0        49173     0          Y       9727 
Brick storinator1:/export/brick6/gv0        49174     0          Y       16920
Brick storinator2:/export/brick6/gv0        49174     0          Y       9733 
Brick storinator1:/export/brick7/gv0        49175     0          Y       16939
Brick storinator2:/export/brick7/gv0        49175     0          Y       9739 
Brick storinator1:/export/brick8/gv0        49176     0          Y       16958
Brick storinator2:/export/brick8/gv0        49176     0          Y       9703 
Brick storinator1:/export/brick9/gv0        49177     0          Y       16977
Brick storinator2:/export/brick9/gv0        49177     0          Y       9713 
Brick storinator1:/export/brick10/gv0       49178     0          Y       16996
Brick storinator2:/export/brick10/gv0       49178     0          Y       9718 
Brick storinator1:/export/brick11/gv0       49179     0          Y       17015
Brick storinator2:/export/brick11/gv0       49179     0          Y       9746 
Brick storinator1:/export/brick12/gv0       49180     0          Y       17034
Brick storinator2:/export/brick12/gv0       49180     0          Y       9792 
Brick storinator1:/export/brick13/gv0       49181     0          Y       17053
Brick storinator2:/export/brick13/gv0       49181     0          Y       9755 
Brick storinator1:/export/brick14/gv0       49182     0          Y       17072
Brick storinator2:/export/brick14/gv0       49182     0          Y       9767 
Brick storinator1:/export/brick15/gv0       49183     0          Y       17091
Brick storinator2:/export/brick15/gv0       N/A       N/A        N       N/A  
Brick storinator1:/export/brick16/gv0       49184     0          Y       17110
Brick storinator2:/export/brick16/gv0       49184     0          Y       9791 
Brick storinator1:/export/brick17/gv0       49185     0          Y       17129
Brick storinator2:/export/brick17/gv0       49185     0          Y       9756 
Brick storinator1:/export/brick18/gv0       49186     0          Y       17148
Brick storinator2:/export/brick18/gv0       49186     0          Y       9766 
Brick storinator1:/export/brick19/gv0       49187     0          Y       17167
Brick storinator2:/export/brick19/gv0       49187     0          Y       9745 
Brick storinator1:/export/brick20/gv0       49188     0          Y       17186
Brick storinator2:/export/brick20/gv0       49188     0          Y       9783 
NFS Server on localhost                     2049      0          Y       17206
Self-heal Daemon on localhost               N/A       N/A        Y       17214
NFS Server on storinator2                   2049      0          Y       9657 
Self-heal Daemon on storinator2             N/A       N/A        Y       9677 

Task Status of Volume gv0
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : 28c733e9-d618-44fc-873f-405d3b29a609
Status               : completed           

Wouldn't you know it, within a week or two of pulling the hardware together and getting gluster installed and configured, a disk dies. Note the dead process for brick15 on server storinator2.

I would like to remove (not replace) the failed brick (and its replica).  (I don't have a spare disk handy, and there's plenty of room on the other bricks.)  But gluster doesn't seem to want to remove a brick if the brick is dead:

[root@storinator1 ~]# gluster volume remove-brick gv0 storinator{1..2}:/export/brick15/gv0 start
volume remove-brick start: failed: Staging failed on storinator2. Error: Found stopped brick storinator2:/export/brick15/gv0

So what do I do?  I can't remove the brick while the brick is bad, but I want to remove the brick *because* the brick is bad.  Bit of a Catch-22.

Hi Phil,

This looks like a bug - please open a new BZ for this issue.  

You can remove these bricks using 
 gluster v remove-brick <vol> <bricks> force

However, that will not copy the data from the bricks that have been removed. You will need to copy the files manually from the brick to the gluster volume mount point.

Thanks,
Nithya

Thanks in advance for any help you can give.

phil

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users