|
Hello all, We have a distributed volume running in 7 hosts and 28 bricks. We've been experiencing some strange behaviors over the time, some of them are solved when the gluster services are restarted, but other problems persist. In the case below, and this one happens many times, some people access the gluster filesystem via Samba. They have a mapped drive in their windows machine. Most of the time things work as expected, but a recurrent issue is that sometimes when directories are created, they get a "permission denied" message. Apparently, the directories
are created. This sometimes happens also when they are renaming directories. And other times they get a message telling them that the files/directories do not exist. There are some logs below.
[2024-06-18 14:02:52.931157 +0000] W [MSGID: 109037] [dht-common.c:4711:dht_getxattr_get_real_filename_cbk] 0-tank-dht: Failed to get real filename. [No such file or
directory] [2024-06-18 14:02:52.934395 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/BallData.mat},
{errno=2}, {error=No such file or directory}] [2024-06-18 14:04:52.221379 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/BallData.mat},
{errno=2}, {error=No such file or directory}] Here we have multiple attempts to rename a file [2024-06-18 14:05:54.992761 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>) [2024-06-18 14:05:55.527004 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>) The message "I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>) " repeated 2 times between [2024-06-18 14:05:55.527004 +0000] and [2024-06-18 14:06:11.390068 +0000] [2024-06-18 14:06:13.417710 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8)
(hash=tank-client-4/cache=tank-client-4) => /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>) There are many messages in the logs like the ones below [2024-06-16 12:28:07.720713 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.4/xlator/performance/md-cache.so(+0x11683) [0x7f998d68f683] -->/lib64/libglusterfs.so.0(dict_get_iatt+0x40)
[0x7f999507bd10] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85) [0x7f99950799b5] ) 0-dict: dict OR key (virt-gf-prestat) is NULL [Invalid argument] [2024-06-16 12:28:29.099386 +0000] W [MSGID: 114061] [client-common.c:3327:client_pre_readdirp_v2] 0-tank-client-0: remote_fd is -1. EBADFD [{gfid=8def2ad4-72fd-49ed-af12-4b1f9878b2a4},
{errno=77}, {error=File descriptor in bad state}] [2024-06-16 12:37:56.662296 +0000] I [MSGID: 109061] [dht-layout.c:641:dht_layout_normalize] 0-tank-dht:
Found anomalies [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {holes=3}, {overlaps=0}] [2024-06-16 12:37:56.663670 +0000] W [MSGID: 109005] [dht-selfheal.c:1064:dht_selfheal_dir_mkdir_cbk] 0-tank-dht:
Healing of path failed [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {errno=2}, {error=No such file or directory}] [2024-06-16 12:37:56.672373 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.4/xlator/cluster/distribute.so(+0x516c6) [0x7f998e3426c6] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3b)
[0x7f999507a5bb] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85) [0x7f99950799b5] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) is NULL [Invalid argument] Did any of you been through this before? The "permission denied" happens randomly. About the renaming of files/directories, sometimes I think that the renames are being done, but the windows client obtains an error and is not informed of the
success of the operation, and then, when they try again to rename, the original filename does not exist anymore, and they get "File not Found". But I'm not sure of this. Thanks. Carlo Rodrigues Cloud & Infra Engineer carlo.rodrigues@xxxxxxxxxxxx T +351 21 319 92 00
Claranet Portugal Todas as informações contidas nesta mensagem eletrónica da Claranet estão abrangidas pelo disclaimer disponível em
https://www.claranet.pt/email-disclaimer All the information contained within this electronic message from Claranet is covered by the disclaimer at https://www.claranet.pt/email-disclaimer-ENG |
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
