hi Guys I'm seeing "Gfid mismatch detected" in the logs but no split brain indicated (4-way replica) Brick swir-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 22 Number of entries in heal pending: 22 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick whale-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 22 Number of entries in heal pending: 22 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick rider-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick dzien:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 10 Number of entries in heal pending: 10 Number of entries in split-brain: 0 Number of entries possibly healing: 0 On swir-ring8: ... The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:bea05f85-a20e-4cbf-9e41-5e351f99b396>/lock_file>, 37b2456f-5216-4679-ac5c-4908b24f895a on USER-HOME-client-15 and ba8f87ed-9bf3-404e-8d67-2631923e1645 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:47:49.034935] and [2020-05-29 21:47:49.079480] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:a8a084b9-6484-48be-bec7-e704cf5ab880>/t>, d7a4ed01-139b-4df3-8070-31bd620a6f15 on USER-HOME-client-15 and d794b6ba-2a1d-4043-bb31-b98b22692763 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:47:49.126173] and [2020-05-29 21:47:49.155432] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:1f686fba-e325-4539-bf48-94b06325d2f4>/Tables.docx>, 344febd8-c89c-4bf3-8ad8-6494c2189c43 on USER-HOME-client-15 and 48d5b12b-03f4-46bf-bed1-9f8f88815615 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:47:49.194061] and [2020-05-29 21:47:49.239896] The message "E [MSGID: 108008] [afr-self-heal-entry.c:257:afr_selfheal_detect_gfid_and_type_mismatch] 0-USER-HOME-replicate-0: Skipping conservative merge on the file." repeated 8 times between [2020-05-29 21:47:49.037812] and [2020-05-29 21:47:49.240423] ... On whale-ring8: ... The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/pcs>, a83d0e5f-ef3a-40ab-be7b-784538d150be on USER-HOME-client-15 and 89af3d31-81fa-4242-b8f7-0f49fd5fe57b on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:45:46.152052] and [2020-05-29 21:45:46.422393] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database>, 81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-15 and 2498a303-8937-43c3-939e-5e1d786b07fa on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:45:46.167704] and [2020-05-29 21:45:46.437702] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/client-state>, fe86c057-c74d-417f-9c2c-6e6eb9778851 on USER-HOME-client-15 and a66f2714-c2a0-4bdc-8786-ad5b93e0e988 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:45:46.144242] and [2020-05-29 21:45:46.442526] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database.1>, 9826d8ad-fecc-4dd7-bc1f-87d0eff23d73 on USER-HOME-client-15 and 81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-13." repeated 3 times between [2020-05-29 21:45:46.162016] and [2020-05-29 21:45:46.476935] ... On rider-ring8: ... 2020-05-29 21:46:53.122929] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied] [2020-05-29 21:46:53.124148] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-4: remote operation failed. Path: <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied] [2020-05-29 21:46:53.133566] I [MSGID: 108026] [afr-self-heal-entry.c:898:afr_selfheal_entry_do] 0-QEMU_VMs-replicate-0: performing entry selfheal on e0121f76-2452-44dc-b1a6-82b46cc9ec79 [2020-05-29 21:46:53.145991] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied] [2020-05-29 21:46:53.147110] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-4: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied] The most recent data I'm 100% certain is on rider-ring8. Any expert could probably quickly suggest how to go about the problem. Would something like manual healing with 'source-brick' or 'latest-mtime' work or is too naive? many thanks, L. ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
