On Thu, May 15, 2014 at 1:16 AM, Vincent Caron <vcaron@xxxxxxxxxxxxx> wrote:
Hello,
sorry for the strange title, but it might be possible that those
symptoms are related...
I'm running a Gluster 3.4.2 volume with 8 bricks in distribute mode.
Easiest problem to diagnose : sometimes a folder goes into a state
where any attempt to create a new file or folder within ends up with
EINVAL (being root or not).
Those errors may persist or not (sometimes it's fixed by itself a
while later), and it seems to be related to this kind of information in
the client's log :
[2014-05-14 19:17:12.937364] I [dht-common.c:623:dht_revalidate_cbk]
0-xxx-prod-dht: mismatching layouts for /uploads/images/5373
[2014-05-14 19:17:12.938960] I
[dht-layout.c:726:dht_layout_dir_mismatch] 0-xxx-prod-dht:
/uploads/images/5373/9c8b - disk layout missing
... where /uploads/images/5373 is a folder which I can list with
readable file, but where creating new files and folder is impossible.
Meanwhile I historically tried to remove a brick from this volume
which has 8 of them. The rebalance operation took ages and I had to
cancel it because I was suffering from performance problems. Now the 8th
brick is shown in the volume status but marked 'decomissioned' in the
internal metadata, and has this property : while all 8 bricks have the
same inode count (roughly 5 millions), only the 7th first have a
balanced block usage with 800 GB, while the 8th has been stuck 270 GB
(the point where I tried to remove it).
It looks expected. The 8th brick is still part of the volume but has
only 'historical' data which is still needed. I'd like to re-add this
brick into my volume, is it as simple as issuing a 'volume add-brick'
and then a 'volume rebalance' ?
If you haven't yet commited the remove-brick operation, you can do remove-brick stop followed by rebalance start operation.
i.e. gluster volume remove-brick <vol> <brick> stop
gluster volume rebalance <vol> start
In any case if there's a way to avoid those bugs, I'd be interested to
hear about it. Thanks for your experience and your help !
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
Thanks & Regards
Shylesh Kumar M
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
