Hi,
We have two cluster running on ganesha NFS backed with glusterfs setup, they are 9.1 and 9.2, and we see those operation error very often, although the mount point is operational, but should I take any action to those error? The mentioned shards are still present in the brick .shard directory, but the head file are already gone. Does it mean it is failing to remove the complete shard chain?
Thank you.
Regards,
Levin
First cluster
[2021-06-07 22:06:14.032672 +0000] E [afr-transaction.c:2751:__afr_eager_lock_handle] (-->/usr/lib64/glusterfs/9.2/xlator/cluster/replicate.so(+0x3dab0) [0x7f65f6b09ab0] -->/usr/lib64/glusterfs/9.2/xlator/cluster/replicate.so(+0x3d590) [0x7f65f6b09590] -->/usr/lib64/glusterfs/9.2/xlator/cluster/replicate.so(+0x3d4ef) [0x7f65f6b094ef] ) 0-: Assertion failed: To attach gdb and coredump, Run the script under "glusterfs/extras/debug/gfcore.py"
[2021-06-08 03:37:28.225245 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.387}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:12.821056 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.393}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:21.443070 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.394}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:21.443117 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-2: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.394}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:32.999235 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.398}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:40.341562 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.399}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:45.375038 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.401}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:52.150104 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.402}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:52.150292 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.402}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:22:27.235500 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-2: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:22:27.235503 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:22:27.235683 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:24:06.220887 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.11}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:30:16.757007 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.21}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:30:27.199865 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.23}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:30:27.199866 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.23}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:37:28.225245 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.387}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:12.821056 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.393}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:21.443070 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.394}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:21.443117 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-2: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.394}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:32.999235 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.398}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:40.341562 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.399}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:45.375038 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.401}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:52.150104 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.402}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 03:41:52.150292 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.402}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:22:27.235500 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-2: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:22:27.235503 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:22:27.235683 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:24:06.220887 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.11}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:30:16.757007 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.21}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:30:27.199865 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.23}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2021-06-08 04:30:27.199866 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.23}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
Second cluster
[2021-06-06 10:00:10.940792 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-0: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:10.940908 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-2: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:10.940980 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-1: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:01:49.884528 +0000] I [MSGID: 133022] [shard.c:3768:shard_delete_shards] 0-hg2-shard: Deleted shards of gfid=752ef763-8529-404b-a817-f8cd7995f07d from backend
[2021-06-06 10:00:11.134620 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-0: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:11.134629 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-1: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:11.134735 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-2: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:10.940908 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-2: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:10.940980 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-1: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:01:49.884528 +0000] I [MSGID: 133022] [shard.c:3768:shard_delete_shards] 0-hg2-shard: Deleted shards of gfid=752ef763-8529-404b-a817-f8cd7995f07d from backend
[2021-06-06 10:00:11.134620 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-0: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:11.134629 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-1: remote operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:11.134735 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-2: remote operation failed. [{errno=13}, {error=Permission denied}]
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
