Re: GlusterFS 9.3 - Replicate Volume (2 Bricks / 1 Arbiter) - Self-healing does not always work

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

>It should be enabled alright, but we have noticed some issues of stale locks (https://github.com/gluster/glusterfs/issues/ {2198, 2211, 2027}) which could prevent self-heal (or any other I/O that takes a blocking lock) from happening.

I have re-enabled cluster.eager-lock.

>But the problem here is different as you noticed. Thorsten needs to find the actual file (`find -samefile`) corresponding to this gfid and see what is the file size, hard-link count etc.) If it is a zero -byte file, then it should be safe to just delete the file and its hardlink from the brick.

I think, here i need your help :) How i can find the file? I only have the gfid from the out of 'gluster volume heal glusterfs-1-volume info' = <gfid:26c5396c-86ff-408d-9cda-106acd2b0768> on Brick 192.168.1.50:/data/glusterfs.

Thanks and regards,
Thorsten

Am So., 31. Okt. 2021 um 07:35 Uhr schrieb Ravishankar N <ravishankar.n@xxxxxxxxxxx>:


On Sat, Oct 30, 2021 at 10:47 PM Strahil Nikolov <hunter86_bg@xxxxxxxxx> wrote:
Hi,

based on the output it seems that for some reason the file was deployed locally but not on the 2-nd brick and the arbiter , which for a 'replica 3 arbiter 1' (a.k.a replica 2 arbiter 1) is strange.

It seems that cluster.eager-lock is enabled as per the virt group: https://github.com/gluster/glusterfs/blob/devel/extras/group-virt.example

@Ravi,

do you think that it should not be enabled by default in the virt group ?

It should be enabled alright, but we have noticed some issues of stale locks (https://github.com/gluster/glusterfs/issues/ {2198, 2211, 2027}) which could prevent self-heal (or any other I/O that takes a blocking lock) from happening. But the problem here is different as you noticed. Thorsten needs to find the actual file (`find -samefile`) corresponding to this gfid and see what is the file size, hard-link count etc.) If it is a zero -byte file, then it should be safe to just delete the file and its hardlink from the brick.

Regards,
Ravi


Best Regards,
Strahil Nikolov



On Sat, Oct 30, 2021 at 16:14, Thorsten Walk
Hi Ravi & Strahil, thanks a lot for your answer!

The file in the path .glusterfs/26/c5/.. only exists at node1 (=pve01). On node2 (pve02) and the arbiter (freya), the file does not exist:



┬[14:35:48] [ssh:root@pve01(192.168.1.50): ~ (700)]
╰─># getfattr -d -m. -e hex  /data/glusterfs/.glusterfs/26/c5/26c5396c-86ff-408d-9cda-106acd2b0768
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/.glusterfs/26/c5/26c5396c-86ff-408d-9cda-106acd2b0768
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.glusterfs-1-volume-client-1=0x000000010000000100000000
trusted.afr.glusterfs-1-volume-client-2=0x000000010000000100000000
trusted.gfid=0x26c5396c86ff408d9cda106acd2b0768
trusted.glusterfs.mdata=0x01000000000000000000000000617880a3000000003b2f011700000000617880a3000000003b2f011700000000617880a3000000003983a635

┬[14:36:49] [ssh:root@pve02(192.168.1.51): /data/glusterfs/.glusterfs/26/c5 (700)]
╰─># ll
drwx------ root root   6B 3 days ago   ./
drwx------ root root 8.0K 6 hours ago  ../

┬[14:36:58] [ssh:root@freya(192.168.1.40): /data/glusterfs/.glusterfs/26/c5 (700)]
╰─># ll
drwx------ root root   6B 3 days ago   ./
drwx------ root root 8.0K 3 hours ago  ../



After this, i have disabled the the option you mentioned:

gluster volume set glusterfs-1-volume cluster.eager-lock off

After that I started another healing process manually. Unfortunately without success.

@Strahil: For your idea with https://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/ i need more time, maybe i can try it tomorrow. I'll be in touch.

Thanks again and best regards,
Thorsten

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux