Client: $ cat /pfs/manu/netbsd/usr/src/tools/mktemp/Makefile cat: /pfs/manu/netbsd/usr/src/tools/mktemp/Makefile: Input/output error Client logs: [2012-06-15 03:32:06.234208] W [afr-common.c:1226:afr_detect_self_heal_by_lookup_status] 0-pfs-replicate-0: split brain detected during lookup of /manu/netbsd/usr/src/tools/mktemp/Makefile. [2012-06-15 03:32:06.234610] I [afr-common.c:1340:afr_launch_self_heal] 0-pfs-replicate-0: background meta-data data gfid self-heal triggered. path: /manu/netbsd/usr/src/tools/mktemp/Makefile, reason: lookup detected pending operations [2012-06-15 03:32:06.236294] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-pfs-replicate-0: no missing files - /manu/netbsd/usr/src/tools/mktemp/Makefile. proceeding to metadata check [2012-06-15 03:32:06.237047] I [afr-self-heal-common.c:994:afr_sh_missing_entries_done] 0-pfs-replicate-0: split brain found, aborting selfheal of /manu/netbsd/usr/src/tools/mktemp/Makefile [2012-06-15 03:32:06.237322] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-pfs-replicate-0: background meta-data data gfid self-heal failed on /manu/netbsd/usr/src/tools/mktemp/Makefile [2012-06-15 03:32:06.238175] W [afr-open.c:213:afr_open] 0-pfs-replicate-0: failed to open as split brain seen, returning EIO [2012-06-15 03:32:06.254921] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 2434279: OPEN() /manu/netbsd/usr/src/tools/mktemp/Makefile => -1 (Input/output error) Brick1: -rw-r--r-- 2 manu manu 165 Dec 8 2002 /export/wd3a/manu/netbsd/usr/src/tools/mktemp/Makefile trusted.gfid 6b db b7 73 cc e7 46 a8 9d fc 96 40 2c 6a fe e8 trusted.afr.pfs-client-0 00 00 00 00 00 00 00 03 00 00 00 00 00 Brick2 (note 0 instead of 3 in trusted.afr.pfs-client-0) -rw-r--r-- 2 manu manu 165 Dec 8 2002 /export/wd3a/manu/netbsd/usr/src/tools/mktemp/Makefile trusted.gfid 6b db b7 73 cc e7 46 a8 9d fc 96 40 2c 6a fe e8 trusted.afr.pfs-client-0 00 00 00 00 00 00 00 00 00 00 00 00 00 I delete the file on brick2 so that self heal can occur. The removed file is indeed created but I still get EIO. Here is client log on self heal: [2012-06-15 03:37:07.717081] I [afr-common.c:1215:afr_detect_self_heal_by_lookup_status] 0-pfs-replicate-0: entries are missing in lookup of /manu/netbsd/usr/src/tools/mktemp/Makefile. [2012-06-15 03:37:07.717459] I [afr-common.c:1340:afr_launch_self_heal] 0-pfs-replicate-0: background meta-data data entry missing-entry gfid self-heal triggered. path: /manu/netbsd/usr/src/tools/mktemp/Makefile, reason: lookup detected pending operations [2012-06-15 03:37:07.719497] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-pfs-replicate-0: path /manu/netbsd/usr/src/tools/mktemp/Makefile on subvolume pfs-client-0 => -1 (No such file or directory) [2012-06-15 03:37:17.571513] I [afr-self-heal-common.c:994:afr_sh_missing_entries_done] 0-pfs-replicate-0: split brain found, aborting selfheal of /manu/netbsd/usr/src/tools/mktemp/Makefile [2012-06-15 03:37:17.571926] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-pfs-replicate-0: background meta-data data entry missing-entry gfid self-heal failed on /manu/netbsd/usr/src/tools/mktemp/Makefile [2012-06-15 03:37:17.572911] W [afr-open.c:213:afr_open] 0-pfs-replicate-0: failed to open as split brain seen, returning EIO [2012-06-15 03:37:17.573365] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 2434373: OPEN() /manu/netbsd/usr/src/tools/mktemp/Makefile => -1 (Input/output error) Nothing in server log at that time. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu@xxxxxxxxxx
