Hi Nithya
Thanks for replying so quickly. It is very much appreciated.
There are lots if " [No space left on device] " errors which I can not understand as there are much space on all of the nodes.
A little bit of background will be useful in this case. I had cluster of seven nodes of varying capacity(73, 73, 73, 46, 46, 46,46 TB) . The cluster was almost 90% full so every node has almost 8 to 15 TB free space. I added two new nodes with 100TB each and ran fix-layout which completed successfully.
After that I started remove-brick operation. I don't think that any point , any of the nodes were 100% full. Looking at my ganglia graph, there is minimum 5TB always available at every node.
I was keeping an eye on remove-brick status and for very long time there was no failures and then at some point these 17000 failures appeared and it stayed like that.
Thanks
Kashif
Let me explain a little bit of background.
On Mon, Feb 4, 2019 at 5:09 AM Nithya Balachandran <nbalacha@xxxxxxxxxx> wrote:
Hi,The status shows quite a few failures. Please check the rebalance logs to see why that happened. We can decide what to do based on the errors.Once you run a commit, the brick will no longer be part of the volume and you will not be able to access those files via the client.Do you have sufficient space on the remaining bricks for the files on the removed brick?Regards,NithyaOn Mon, 4 Feb 2019 at 03:50, mohammad kashif <kashif.alig@xxxxxxxxx> wrote:_______________________________________________HiI have a pure distributed gluster volume with nine nodes and trying to remove one node, I rangluster volume remove-brick atlasglust nodename:/glusteratlas/brick007/gv0 startIt completed but with around 17000 failuresNode Rebalanced-files size scanned failures skipped status run time in h:m:s--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------nodename 4185858 27.5TB 6746030 17488 0 completed 405:15:34I can see that there is still 1.5 TB of data on the node which I was trying to remove.I am not sure what to do now? Should I run remove-brick command again so the files which has been failed can be tried again?or should I run commit first and then try to remove node again?Please advise as I don't want to remove files.ThanksKashif
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
