"gfid differs on subvolume"

[Gambit15 <dougti+gluster@xxxxxxxxx>]

Hi Guys,

 I've got a distributed replica 2+1 (rep 3 arbiter 1) cluster, and it appears a shard has been assigned different GFIDs on each replica set.

===============
[2018-11-29 10:05:12.035422] W [MSGID: 109009] [dht-common.c:2148:dht_lookup_linkfile_cbk] 0-data-novo-dht: /.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid different on data file on data-novo-replicate-1, gfid local = 00000000-0000-0000-0000-000000000000, gfid node = 492d52d6-e3d1-4ed4-918d-9cdab7a135e0
[2018-11-29 10:05:12.036120] W [MSGID: 109009] [dht-common.c:1887:dht_lookup_everywhere_cbk] 0-data-novo-dht: /.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid differs on subvolume data-novo-replicate-1, gfid local = c7f6cc63-ae40-4d1a-aa6f-fe97f7912036, gfid node = 492d52d6-e3d1-4ed4-918d-9cdab7a135e0
[2018-11-29 10:05:12.036159] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 0-data-novo-shard: Lookup on shard 1846 failed. Base file gfid = a46fd27c-5aa6-4fc8-b8e1-c097065e7096 [Stale file handle]
[2018-11-29 10:05:12.036184] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 6916126: READ => -1 gfid=a46fd27c-5aa6-4fc8-b8e1-c097065e7096 fd=0x7f12fa57f06c (Stale file handle)
===============

==== FIRST REPLICA SET (v0-v2) ====

v0:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v0:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

# file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036
trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100

v1:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v1:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

# file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036
trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100

v2:~$ ls -l /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

---------T. 2 root root         0 Oct 17 11:28 /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v2:~$ getfattr -d -m . -e hex /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

# file: gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036
trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100

==== SECOND REPLICA SET (v0-v2) ====

v2:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

-rw-rw----. 2 root root 536870912 Nov 27 14:15 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v2:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

# file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x02000000000000005b4a3790000750d5
trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0

v3:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

-rw-rw----. 2 root root 536870912 Nov 27 14:15 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v3:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

# file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x04000000000000005bda0242000c7cdd
trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0

v0:~$ ls -l /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

-rw-rw----. 2 root root 0 Oct 17 11:28 /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v0:~$ getfattr -d -m . -e hex /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

# file: gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x03000000000000005bda02410001b61a
trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0

=====================================

So the shard has GFID " 0xc7f6cc63ae404d1aaa6ffe97f7912036" on the first replica set, and GFID "0x492d52d6e3d14ed4918d9cdab7a135e0" on the second replica set.

How do I go about correcting this?

Many thanks,

 Doug

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users