Hi,
We are facing a weird scenario where on the clients the glusterfs volume mounts successfully but the size on the client is less than the total volume size. A few bricks are missing and hence the size as seen on the client is incorrect and we face issues of missing files. This happens randomly and we are unable to reproduce it. The gluster volume status shows all bricks up and running.
We are running distributed glusterfs version 9.3 with 13 nodes having 50 bricks divided amongst them. We are mounting it on the client with the below entry in fstab
"10.132.2.101:/datavol /data glusterfs ro,_netdev 0 0" where 10.132.2.101 is one of the nodes and datavol is the volume name.
We are facing a weird scenario where on the clients the glusterfs volume mounts successfully but the size on the client is less than the total volume size. A few bricks are missing and hence the size as seen on the client is incorrect and we face issues of missing files. This happens randomly and we are unable to reproduce it. The gluster volume status shows all bricks up and running.
We are running distributed glusterfs version 9.3 with 13 nodes having 50 bricks divided amongst them. We are mounting it on the client with the below entry in fstab
"10.132.2.101:/datavol /data glusterfs ro,_netdev 0 0" where 10.132.2.101 is one of the nodes and datavol is the volume name.
Any help would be appreciated. In the client glusterfs log we see the below prints in case these help to figure out the issue.
[2025-01-28 11:00:11.404092 +0000] W [MSGID: 109222] [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
[2025-01-28 11:00:11.405029 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea) [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) [0x7f63
dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e) [0x7f63dd12f29e] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) is NULL [Invalid argument]
[2025-01-28 11:00:11.406595 +0000] I [MSGID: 109061] [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies [{path=/code}, {gfid=949ca970-0b14-4fed-af73-d9ea216905f1}, {holes=1}, {overlaps=0}]
[2025-01-28 11:00:11.406731 +0000] W [MSGID: 109222] [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory selfheal failed [{path=/code}, {subvol-down=1}, {Not-fixing}, {gfid=949ca970-0b14-4fed-a
f73-d9ea216905f1}]
[2025-01-28 11:00:11.408356 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea) [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) [0x7f63
dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e) [0x7f63dd12f29e] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) is NULL [Invalid argument]
[2025-01-28 11:00:11.409546 +0000] I [MSGID: 109061] [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies [{path=/code/summary}, {gfid=c96f0e78-dd0e-4c61-89a3-f94ca1b502c4}, {holes=1}, {overlap
s=0}]
[2025-01-28 11:00:11.409605 +0000] W [MSGID: 109222] [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory selfheal failed [{path=/code/summary}, {subvol-down=1}, {Not-fixing}, {gfid=c96f0e78-dd0
e-4c61-89a3-f94ca1b502c4}]
[2025-01-28 11:00:28.910500 +0000] W [MSGID: 109222] [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
[2025-01-28 11:00:28.911125 +0000] I [MSGID: 109061] [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies [{path=/}, {gfid=00000000-0000-0000-0000-000000000001}, {holes=1}, {overlaps=0}]
[2025-01-28 11:00:28.911167 +0000] W [MSGID: 109222] [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
[2025-01-28 11:00:40.359587 +0000] W [MSGID: 109222] [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
[2025-01-28 11:00:40.375728 +0000] W [MSGID: 109011] [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed subvolume [{hash-value=0x5877673a}]
[2025-01-28 11:00:40.377646 +0000] W [MSGID: 109011] [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed subvolume [{hash-value=0x5877673a}]
[2025-01-28 11:00:40.428513 +0000] W [MSGID: 109011] [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed subvolume [{hash-value=0x599f6a37}]
Regards,
Shreyansh Shah
Shreyansh Shah
AlphaGrep Securities Pvt. Ltd.
------- Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-devel
