Greetings,
I have a problem in replicating an existing gluster volume from single brick setup to two brick setup. Background of the problem is following:
OS: Ubuntu 14.04
Gluster version (from gluster repos): glusterfs 3.7.14 built on Aug 1 2016 16:57:28
1. I had a replication setup consisting of two Gluster bricks (srv100, srv102), and three volumes (gv0, gv100).
2. I had to completely rebuild raid/disks of one of the bricks (srv100) due to hardware failure. I did it by doing following on the faulty node:
2.1 Removed the failed brick from replication setup (reduced replica count to 1 from 2, and detached the node). I executed following commands on the *good* brick.
sudo gluster volume remove-brick gv100 replica 1 srv100:/pool01/gfs/brick1/
gv100 force sudo gluster volume remove-brick gv0 replica 1 srv100:/pool01/gfs/brick1/gv0 force
sudo gluster vol info #make sure the faulty node bricks are not listed, and brick count is 1 for each volume
sudo gluster peer detach srv100 force
sudo gluster peer status # --> OK, only one node/brick
2.2 Stopped glusterd, killed all gluster processes
2.3 Replaced HDs, and recreated raid. This means all GlusterFS data relevant directories were lost on the faulty-brick (srv100), while GlusterFS service installation and config files were untouched (including host name and IP address).
2.4 After rebuilding, I created volume directories on the rebuilt-node
2.5 Then I started gluster service, and added the node back to gluster cluster. Peer status is ok (in cluster)
2.6 Then I attempted to replicate one of the existing volume (gv0), and *there* came the problem. The replication could not be setup properly, and gave following error
sudo gluster volume add-brick gv0 replica 2 srv100:/pool01/gfs/brick1/gv0
volume add-brick: failed: Staging failed on srv100. Please check log file for details.
The relevant gluster log file says
[2016-08-25 12:32:29.499708] I [MSGID: 106499] [glusterd-handler.c:4267:__
glusterd_handle_status_volume] 0-management: Received status volume req for volume gv-temp [2016-08-25 12:32:29.501881] E [MSGID: 106301] [glusterd-syncop.c:1274:gd_
stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gv-temp is not started [2016-08-25 12:32:29.505033] I [MSGID: 106499] [glusterd-handler.c:4267:__
glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2016-08-25 12:32:29.508585] E [MSGID: 106153] [glusterd-syncop.c:113:gd_
collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv0 does not exist [2016-08-25 12:32:29.511062] I [MSGID: 106499] [glusterd-handler.c:4267:__
glusterd_handle_status_volume] 0-management: Received status volume req for volume gv100 [2016-08-25 12:32:29.514556] E [MSGID: 106153] [glusterd-syncop.c:113:gd_
collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv100 does not exist [2016-08-25 12:33:15.865773] I [MSGID: 106499] [glusterd-handler.c:4267:__
glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2016-08-25 12:33:15.869441] E [MSGID: 106153] [glusterd-syncop.c:113:gd_
collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv0 does not exist [2016-08-25 12:33:15.872630] I [MSGID: 106499] [glusterd-handler.c:4267:__
glusterd_handle_status_volume] 0-management: Received status volume req for volume gv100 [2016-08-25 12:33:15.876199] E [MSGID: 106153] [glusterd-syncop.c:113:gd_
collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv100 does not exist [2016-08-25 12:34:14.716735] I [MSGID: 106482] [glusterd-brick-ops.c:442:__
glusterd_handle_add_brick] 0-management: Received add brick req [2016-08-25 12:34:14.716787] I [MSGID: 106062] [glusterd-brick-ops.c:494:__
glusterd_handle_add_brick] 0-management: replica-count is 2 [2016-08-25 12:34:14.716809] I [MSGID: 106447] [glusterd-brick-ops.c:240:gd_
addbr_validate_replica_count] 0-management: Changing the type of volume gv0 from 'distribute' to 'replica' [2016-08-25 12:34:14.720133] E [MSGID: 106153] [glusterd-syncop.c:113:gd_
collate_errors] 0-glusterd: Staging failed on srv100. Please check log file for details.
3. I tried to create a new replicated volume (gv-temp) over the nodes à it is created and replicated. It is only that the existing volume I cannot replicate again!
4. I also observed that /var/lib/glusterd/vols directory on the rebuilt node contains directory for the newly created volume (gv-temp), and no existing volumes (gv100, gv0)
*Questions:*
a. How to re-replicate the exiting volume, for which I set the replica count to 1 (see point 2.1)?
b. Is there a “glusterfs” way to create missing volume directories (under /var/lib/glusterd/vols) on the re-built node (see point 4)?
c. Any other pointers, hints?
Thanks.
Kind regards,
JAsghar
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
--
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
