Hello ; I found many self-heal triggered log in every 10 minutes. Only one file , it 's gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f. Heal-failed and split-brain have not display anything. Does any problem in this file ? GlusterFS config: OS: 2.6.32-220.17.1.el6.x86_64 Scientific Linux release 6.2 (Carbon) # rpm -qa | grep glusterfs glusterfs-3.3.0-2.el6.x86_64 glusterfs-devel-3.3.0-2.el6.x86_64 glusterfs-fuse-3.3.0-2.el6.x86_64 glusterfs-geo-replication-3.3.0-2.el6.x86_64 glusterfs-rdma-3.3.0-2.el6.x86_64 glusterfs-server-3.3.0-2.el6.x86_64 glusterfs-debuginfo-3.3.0-2.el6.x86_64 gluster> volume info Volume Name: gvol1 Type: Distributed-Replicate Volume ID: a7d8ffdf-7296-404b-aeab-824ee853ec59 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 172.30.1.125:/export/data00 Brick2: 172.30.1.125:/export/data01 Brick3: 172.30.1.125:/export/data10 Brick4: 172.30.1.125:/export/data11 Options Reconfigured: features.limit-usage: /source:500GB features.quota: on performance.cache-refresh-timeout: 30 performance.io-thread-count: 32 nfs.disable: off cluster.min-free-disk: 5% performance.cache-size: 128MB gluster volume heal gvol1 info Heal operation on volume gvol1 has been successful Brick 172.30.1.125:/export/data00 Number of entries: 0 Brick 172.30.1.125:/export/data01 Number of entries: 0 Brick 172.30.1.125:/export/data10 Number of entries: 1 /fs126/Graphite-monitor_vdb.qcow2 Brick 172.30.1.125:/export/data11 Number of entries: 1 /fs126/Graphite-monitor_vdb.qcow2 # gluster volume heal gvol1 info heal-failed Heal operation on volume gvol1 has been successful Brick 172.30.1.125:/export/data00 Number of entries: 0 Brick 172.30.1.125:/export/data01 Number of entries: 0 Brick 172.30.1.125:/export/data10 Number of entries: 0 Brick 172.30.1.125:/export/data11 Number of entries: 0 gluster volume heal gvol1 info split-brain Heal operation on volume gvol1 has been successful Brick 172.30.1.125:/export/data00 Number of entries: 0 Brick 172.30.1.125:/export/data01 Number of entries: 0 Brick 172.30.1.125:/export/data10 Number of entries: 0 Brick 172.30.1.125:/export/data11 Number of entries: 0 Log detail: [2012-07-12 09:13:11.666417] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:13:11.666998] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:13:11.667132] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:13:11.667141] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:13:11.667589] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:13:11.668581] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:13:11.669322] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:13:11.669775] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:13:11.670408] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.770994] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:23:11.771358] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:23:11.771416] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:23:11.771898] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:23:11.772059] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:23:11.773074] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.773686] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.774094] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.774652] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.874474] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:33:11.874919] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:33:11.874978] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: d9b0c350-33ba-4090-ab08-f91f30dd661f: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:33:11.875505] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:33:11.875676] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:33:11.876613] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.877244] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.877646] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.878191] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.971727] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:43:11.972004] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:43:11.972066] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:43:11.972635] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:43:11.972924] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:43:11.973885] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.974451] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.974965] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.975772] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> -- Best Regards Homer Li
