During a rebalance, errors occured for some files resulting in duplicate files on the volume.
[2022-03-14 16:24:47.957556 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/label-full-defect-qa-20191107.json is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/label-full-defect-qa-20191107.json [Operation not supported]
With the corresponding messages on the bricks logs:
/var/log/glusterfs/bricks/bricks-14c2df57-2751-4567-8849-5ae9786eb264-brick.log.1:[2022-03-14 16:24:47.687763 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-14c2df57-2751-4567-8849-5ae9786eb264-brick.log.1:[2022-03-14 16:24:48.567493 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-96870e54-4a0b-4ada-9339-fcde096c346a-brick.log.1:[2022-03-14 16:24:47.687588 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-96870e54-4a0b-4ada-9339-fcde096c346a-brick.log.1:[2022-03-14 16:24:48.567201 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-3ba68667-bd43-45e0-8e8e-b2538dcfeb96-brick.log.1:[2022-03-14 16:24:47.687312 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-3ba68667-bd43-45e0-8e8e-b2538dcfeb96-brick.log.1:[2022-03-14 16:24:48.566871 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-0e213dec-2b72-4dd1-94aa-85d94e5a3639-brick.log.1:[2022-03-14 16:24:47.686811 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-0e213dec-2b72-4dd1-94aa-85d94e5a3639-brick.log.1:[2022-03-14 16:24:48.566324 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14 16:24:47.687322 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14 16:24:48.567089 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-f84410a1-636b-48e5-b1c2-ff1ff87504a0-brick.log.1:[2022-03-14 16:24:47.687433 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-f84410a1-636b-48e5-b1c2-ff1ff87504a0-brick.log.1:[2022-03-14 16:24:48.567062 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
My understanding is that this xattr (glusterfs.inodelk-dom-count) could not be written (because it doesnt start with trusted, user, system, or security) and that actually these should not be attempted as this is filtered:
https://github.com/gluster/glusterfs/blob/v9.3/xlators/storage/posix/src/posix-helpers.c#L1853
I'm not sure how else the execution could have got to that point.
We have noticed some issues with the volume previously. Some directories had inconsistent gfids (they were consistent across the subvolumes, but about 15 of the subvolumes differed for some directories)
Any help appreciated
More info here https://github.com/gluster/glusterfs/issues/3309
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
