On 05/31/2016 08:19 PM, Johan Moreels wrote:
Dear developers,
I would like to thank you for the great job in developing gluster for the
open-source community and I value the various improvements in the successive
versions from 3.4 up to 3.7 that I used.
However, as I already mentionned in my older posts, gluster has an issue to
handle high loads, i.e. normal use, when multiple clients try to access its
volume. This issue combined to the fact that in the past I used versions
suffering from serious memory leaks (for example 3.6.9 which despite the fact
that the issues were known has not been explicitly discouraged to use) which
results in brick processes being killed by OOM kernel process. It results in
about 40,000 files currently in split-brain conditions, while gluster was used
normally.
The current advise from the developers to solve the split-brain cases is
to fix them manually which in my case is just not possible. This is why I plea
that you urgently decide to include the CLI tools that allow to
list according to the type of split-brain (either files diff and/or attributes
diff and if attributes, what kind of attributes, i.e. quota) and a automatic
way to fix a specific type of split-brain. As an example, if split-brain is
detected due to mismatch between quota attributes for many files, the CLI
should allow to invalidate these attributes and refresh them at once for all
these files. As I already said, it is nice to add new features, but it is a
little bit problematic if some basic management/repair tools are lacking.
I must really stress that it is not the task of users to develop
strategies/scripts to fix split-brain issues (additional amount of work to our
normal work) but the developers, or am I missing something here ?
Hi Johan,
Recent versions of gluster (3.7.x, IIRC) have a CLI that you can use to
fix split-brains based on various policies instead of messing around
with the extended attributes yourself.
You can also resolve split-brains from the mount point using a virtual
setfattr command. Both these methods are detailed in [1]. The soon to be
released glusterfs 3.8
also has a new volume option [2] which can be set to automatically pick
a source based on various policies.
And finally, could you get rid of any gfid in the logs and heal CLI output but
instead give the file or directory, because the gfid alone is simply useless
to find the file/dir on multi-terabyte volume/bricks in a decent amount of
time ?
We print the gfid only if the file name is not available from the brick
process (because the brick just came back up and there was no lookup
made on the brick from a client. i.e. the inode/dentry information is
not yet populated in brick process).
The CLI referred to in [1] also accepts gfids to automate split-brain
resolution. There are ways [3] to find the file name from gfid without
crawling the filesystem for an inode number match.
Hope this helps,
Ravi
[1]https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md
[2]
http://review.gluster.org/#/c/14554/2/done/GlusterFS+3.8/automagic-unsplit-brain.md
[3] https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/
Thanks,
Alessandro.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
