On 08/13/2013 03:43 AM, Cool wrote:
> remove-brick in 3.4.0 seems removing wrong bricks, can someone help to
> review the environment/steps to see if I did anything stupid?
>
> setup - Ubuntu 12.04LTS on gfs11 and gfs12, with following packages
> from ppa, both nodes have 3 xfs partitions sdb1, sdc1, sdd1:
> ii glusterfs-client 3.4.0final-ubuntu1~precise1
> clustered file-system (client package)
> ii glusterfs-common 3.4.0final-ubuntu1~precise1
> GlusterFS common libraries and translator modules
> ii glusterfs-server 3.4.0final-ubuntu1~precise1
> clustered file-system (server package)
>
> step to reproduce the problem:
> 1. create volume gfs_v0 in replica 2 with gfs11:/sdb1 and gfs12:/sdb1
> 2. add-brick gfs11:/sdc1 and gfs12:/sdc1
> 3. add-brick gfs11:/sdd1 and gfs12:/sdd1
> 4. rebalance to make files distributed to all three pair of disks
> 5. remove-brick gfs11:/sdd1 and gfs12:/sdd1 start, files on
> ***/sdc1*** are migrating out
> 6. remove-brick commit led to data loss in gfs_v0
>
> If between step 5 and 6 I initiate a remove-brick targeting /sdc1,
> then after commit I would not lose anything since all data will be
> migrated back to /sdb1.
>
You should ensure that a 'remove-brick start ' has completed and then
commit it before initiating the second one. The correct way to do this
would be:
5. # gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 start
6. Check that the data migration has been completed using the status
command:
# gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 status
7. #gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 commit
8. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 start
9. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 status
10. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 commit
This would leave you with the original replica 2 volume that you had
begun with. Hope this helps.
Note:
The latest version of glusterfs has the check that prevents a second
remove-brick operation until the first one has been committed.
(You would receive a message thus : "volume remove-brick start: failed:
An earlier remove-brick task exists for volume <volname>. Either commit
it or stop it before starting a new task." )
-Ravi
> -C.B.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
