Actually we just discovered that the heal info command was returning different things when executed on the different nodes of our 3-replica setup.
When we execute it on node2 we did not see the split brain reported "/" but if I execute it on node0 and node1 I am seeing:
x@gfs-vm001:~$ sudo gluster volume heal gv0 info | tee heal-info
Brick gfs-vm000:/gluster/brick/brick0
<gfid:81289110-867b-42ff-ba3b-1373a187032b>
/ - Is in split-brain
Status: Connected
Number of entries: 2
Brick gfs-vm001:/gluster/brick/brick0
/ - Is in split-brain
<gfid:81289110-867b-42ff-ba3b-1373a187032b>
Status: Connected
Number of entries: 2
Brick gfs-vm002:/gluster/brick/brick0
/ - Is in split-brain
Status: Connected
Number of entries: 1
I ran getfattr -d -m . -e hex /gluster/brick/brick0 on all three nodes and I am seeing node2 has slightly different attr:
node0:
sudo getfattr -d -m . -e hex /gluster/brick/brick0
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick/brick0
trusted.afr.gv0-client-2=0x000000000000000100000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
node1:
sudo getfattr -d -m . -e hex /gluster/brick/brick0
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick/brick0
trusted.afr.gv0-client-2=0x000000000000000100000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
node2:
sudo getfattr -d -m . -e hex /gluster/brick/brick0
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick/brick0
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.gv0-client-0=0x000000000000000200000000
trusted.afr.gv0-client-1=0x000000000000000200000000
trusted.afr.gv0-client-2=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
Where do I go from here? Thanks
On Tue, Jul 3, 2018 at 11:54 AM, Anh Vo <vtqanh@xxxxxxxxx> wrote:
I am trying to mount nfs to gluster volume and got mount.nfs failure. Looking at nfs.log I am seeing these entriesHeal info does not show the mentioned gfid ( 00000000-0000-0000-0000-000000000001 ) being in split-brain.[2018-07-03 18:16:27.694953] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: / => (XID: c3ac3cc5, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error)) [2018-07-03 18:16:28.204685] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: / => (XID: c4ac3cc5, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error)) The message "E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-gv0-replicate-0: Failing STAT on gfid 00000000-0000-0000-0000- 000000000001: split-brain observed. [Input/output error]" repeated 2 times between [2018-07-03 18:16:27.694903] and [2018-07-03 18:17:02.310689] [2018-07-03 18:17:02.310722] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: / => (XID: 2a6f2526, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error)) [2018-07-03 18:17:02.628990] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-gv0-replicate-0: Failing STAT on gfid 00000000-0000-0000-0000- 000000000001: split-brain observed. [Input/output error] [2018-07-03 18:17:02.629023] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: / => (XID: 2b6f2526, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error)) [2018-07-03 18:17:00.398601] I [MSGID: 108031] [afr-common.c:2458:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-2 [2018-07-03 18:17:01.666671] W [MSGID: 108027] [afr-common.c:2821:afr_discover_done] 0-gv0-replicate-0: no read subvols for / [2018-07-03 18:51:43.509385] W [MSGID: 108027] [afr-common.c:2821:afr_discover_done] 0-gv0-replicate-0: no read subvols for / [2018-07-03 18:51:43.936826] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-gv0-replicate-0: Failing STAT on gfid 00000000-0000-0000-0000- 000000000001: split-brain observed. [Input/output error] [2018-07-03 18:51:43.936868] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: / => (XID: 19b1731e, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error)) [2018-07-03 18:51:44.278901] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: / => (XID: 1ab1731e, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
