| Thank you Karthik,
I have run this for all files (see example below) and it says the file is not in split-brain:
sudo gluster volume heal storage2 split-brain latest-mtime /dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59 Healing /dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59 failed: File not in split-brain. Volume heal failed.
- Kindest regards, Milos Cuculovic IT Manager --- MDPI AG Postfach, CH-4020 Basel, Switzerland Office: St. Alban-Anlage 66, 4052 Basel, Switzerland Tel. +41 61 683 77 35 Fax +41 61 302 89 18 Email: cuculovic@xxxxxxxxSkype: milos.cuculovic.mdpi
Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.
Hi Milos,
Thanks for the logs and the getfattr output. From the logs I can see that there are 6 entries under the directory "/data/data-cluster/dms/final_archive" named 41be9ff5ec05c4b1c989c6053e709e59 5543982fab4b56060aa09f667a8ae617 a8b7f31775eebc8d1867e7f9de7b6eaf c1d3f3c2d7ae90e891e671e2f20d5d4b e5934699809a3b6dcfc5945f408b978b e7cdc94f60d390812a5f9754885e119e which are having gfid mismatch, so the heal is failing on this directory.
You can use the CLI option to resolve these files from gfid mismatch. You can use any of the 3 methods available: 1. bigger-file gluster volume heal <VOLNAME> split-brain bigger-file <FILE>
2. latest-mtime gluster volume heal <VOLNAME> split-brain latest-mtime <FILE>
3. source-brick gluster volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME> <FILE>
where <FILE> must be absolute path w.r.t. the volume, starting with '/'. If all those entries are directories then go for either latest-mtime/source-brick option. After you resolve all these gfid-mismatches, run the "gluster volume heal <volname>" command. Then check the heal info and let me know the result.
Regards, Karthik Sure, thank you for following up.
About the commands, here is what I see:
brick1: ————————————————————————————————————— sudo gluster volume heal storage2 info Brick storage3:/data/data-cluster <gfid:256ca960-1601-4f0d-9b08-905c6fd52326> <gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> /dms/final_archive - Possibly undergoing heal
Status: Connected Number of entries: 3
Brick storage4:/data/data-cluster <gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> /dms/final_archive - Possibly undergoing heal
Status: Connected Number of entries: 2 ————————————————————————————————————— sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive getfattr: Removing leading '/' from absolute path names # file: data/data-cluster/dms/final_archive trusted.afr.dirty=0x000000000000000000000000 trusted.afr.storage2-client-1=0x000000000000000000000010 trusted.gfid=0x16c6a1e2b3fe4851972b998980097a87 trusted.glusterfs.dht=0x000000010000000000000000ffffffff trusted.glusterfs.dht.mds=0x00000000 ————————————————————————————————————— stat /data/data-cluster/dms/final_archive File: '/data/data-cluster/dms/final_archive' Size: 3497984 Blocks: 8768 IO Block: 4096 directory Device: 807h/2055d Inode: 26427748396 Links: 72123 Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data) Access: 2018-10-09 04:22:40.514629044 +0200 Modify: 2019-03-21 11:55:37.382278863 +0100 Change: 2019-03-21 11:55:37.382278863 +0100 Birth: - ————————————————————————————————————— —————————————————————————————————————
brick2: ————————————————————————————————————— sudo gluster volume heal storage2 info Brick storage3:/data/data-cluster <gfid:256ca960-1601-4f0d-9b08-905c6fd52326> <gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> /dms/final_archive - Possibly undergoing heal
Status: Connected Number of entries: 3
Brick storage4:/data/data-cluster <gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> /dms/final_archive - Possibly undergoing heal
Status: Connected Number of entries: 2 ————————————————————————————————————— sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive getfattr: Removing leading '/' from absolute path names # file: data/data-cluster/dms/final_archive trusted.afr.dirty=0x000000000000000000000000 trusted.afr.storage2-client-0=0x000000000000000000000001 trusted.gfid=0x16c6a1e2b3fe4851972b998980097a87 trusted.glusterfs.dht=0x000000010000000000000000ffffffff trusted.glusterfs.dht.mds=0x00000000 ————————————————————————————————————— stat /data/data-cluster/dms/final_archive File: '/data/data-cluster/dms/final_archive' Size: 3497984 Blocks: 8760 IO Block: 4096 directory Device: 807h/2055d Inode: 13563551265 Links: 72124 Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data) Access: 2018-10-09 04:22:40.514629044 +0200 Modify: 2019-03-21 11:55:46.382565124 +0100 Change: 2019-03-21 11:55:46.382565124 +0100 Birth: - —————————————————————————————————————
Hope this helps.
- Kindest regards, Milos Cuculovic IT Manager --- MDPI AG Postfach, CH-4020 Basel, Switzerland Office: St. Alban-Anlage 66, 4052 Basel, Switzerland Tel. +41 61 683 77 35 Fax +41 61 302 89 18 Email: cuculovic@xxxxxxxxSkype: milos.cuculovic.mdpi
Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.
Can you attach the "glustershd.log" file which will be present under "/var/log/glusterfs/" from both the nodes and the "stat" & "getfattr -d -m . -e hex <file-path-on-brick>" output of all the entries listed in the heal info output from both the bricks? Thanks Karthik!
I was trying to find some resolution methods from [2] but unfortunately none worked (I can explain what I tried if needed).
I guess the volume you are talking about is of type replica-2 (1x2).
That’s correct, aware of the arbiter solution but still didn’t took time to implement.
From the info results I posted, how to know in which situation I am. No files are mentioned in spit brain, only directories. One brick has 3 entries and one two entries.
sudo gluster volume heal storage2 info [sudo] password for sshadmin: Brick storage3:/data/data-cluster <gfid:256ca960-1601-4f0d-9b08-905c6fd52326> <gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> /dms/final_archive - Possibly undergoing heal
Status: Connected Number of entries: 3
Brick storage4:/data/data-cluster <gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> /dms/final_archive - Possibly undergoing heal
Status: Connected Number of entries: 2
- Kindest regards, Milos Cuculovic IT Manager --- MDPI AG Postfach, CH-4020 Basel, Switzerland Office: St. Alban-Anlage 66, 4052 Basel, Switzerland Tel. +41 61 683 77 35 Fax +41 61 302 89 18 Email: cuculovic@xxxxxxxxSkype: milos.cuculovic.mdpi
Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.
Hi,
Note: I guess the volume you are talking about is of type replica-2 (1x2). Usually replica 2 volumes are prone to split-brain. If you can consider converting them to arbiter or replica-3, they will handle most of the cases which can lead to slit-brains. For more information see [1].
Resolving the split-brain: [2] talks about how to interpret the heal info output and different ways to resolve them using the CLI/manually/using the favorite-child-policy. If you are having entry split brain, and is a gfid split-brain (file/dir having different gfids on the replica bricks) then you can use the CLI option to resolve them. If a directory is in gfid split-brain in a distributed-replicate volume and you are using the source-brick option please make sure you use the brick of this subvolume, which has the same gfid as that of the other distribute subvolume(s) where you have the correct gfid, as the source. If you are having a type mismatch then follow the steps in [3] to resolve the split-brain.
HTH, Karthik I was now able to catch the split brain log:
sudo gluster volume heal storage2 info Brick storage3:/data/data-cluster <gfid:256ca960-1601-4f0d-9b08-905c6fd52326> <gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> /dms/final_archive - Is in split-brain
Status: Connected Number of entries: 3
Brick storage4:/data/data-cluster <gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> /dms/final_archive - Is in split-brain
Status: Connected Number of entries: 2
Since 24h, after upgrading from 4.0 to 4.1.7 one of the servers, the heal shows this:
sudo gluster volume heal storage2 info
Brick storage3:/data/data-cluster
<gfid:256ca960-1601-4f0d-9b08-905c6fd52326>
<gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6>
/dms/final_archive - Possibly undergoing heal
Status: Connected
Number of entries: 3
Brick storage4:/data/data-cluster
<gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0>
/dms/final_archive - Possibly undergoing heal
Status: Connected
Number of entries: 2
The same files stay there. From time to time the status of the /dms/final_archive is in split brain at the following command shows:
sudo gluster volume heal storage2 info split-brain
Brick storage3:/data/data-cluster
/dms/final_archive
Status: Connected
Number of entries in split-brain: 1
Brick storage4:/data/data-cluster
/dms/final_archive
Status: Connected
Number of entries in split-brain: 1
How to know the file who is in split brain? The files in /dms/final_archive are not very important, fine to remove (ideally resolve the split brain) for the ones that differ.
I can only see the directory and GFID. Any idea on how to resolve this situation as I would like to continue with the upgrade on the 2nd server, and for this the heal needs to be done with 0 entries in sudo gluster volume heal storage2 info
Thank you in advance, Milos.
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
|
