Re: Gluster v6.6 replica 2 arbiter 1 - gfid on arbiter is different

[Strahil Nikolov <hunter86_bg@xxxxxxxxx>]

It seems that whenever I reboot a gluster node , I got this problem - so it's not an arbiter issue.

Obviously there is something wrong with v6.6 ,as I never had such issues with v6.5 .

Any ideas where should I start this up ?

Best Regards,

Strahil Nikolov

В сряда, 13 ноември 2019 г., 22:23:38 ч. Гринуич+2, Strahil Nikolov <hunter86_bg@xxxxxxxxx> написа:

Hello Community,

I had a power issue last night and all the systems went down. When they powered up the arbiter was offline (network issue) which was resolved and heal passed for almost all files but 2 per volume.

The strange thing is that the arbiter's copy is older and the 'trusted.gfid' is different on the arbiter. In which cases this can happen ?

I guess the heal is not passing ,as the 'trusted.gfid' is different while 'trusted.gfid2path.<long_string>' is the same.

Here is the output of getfattr and I need to realise if this is a possible situation or it is a bug that needs reporting:

Brick1:

[root@ovirt1 ~]# getfattr -d -m . -e hex /gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b
b2.meta
getfattr: Removing leading '/' from absolute path names
# file: gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.data_fast-client-2=0x000000020000000200000000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x2408e73df137431e99c663783b8bb356
trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746
1
trusted.glusterfs.mdata=0x010000000000000000000000005dcc1eae0000000023c985d9000000005dcc1eae000000001f4c18bf000000005dcc1eae000000001ed22d12
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000

Brick2:

[root@ovirt2 ~]# getfattr -d -m . -e hex /gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b
b2.meta
getfattr: Removing leading '/' from absolute path names
# file: gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.data_fast-client-2=0x000000020000000200000000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x2408e73df137431e99c663783b8bb356
trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746
1
trusted.glusterfs.mdata=0x010000000000000000000000005dcc1eae0000000023c985d9000000005dcc1eae000000001f4c18bf000000005dcc1eae000000001ed22d12
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000

Arbiter:

[root@ovirt3 ~]# getfattr -d -m . -e hex /gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b
b2.meta
getfattr: Removing leading '/' from absolute path names
# file: gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48c9c4ebe75c41f7ac45ea4943e9a65a
trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746
1
trusted.glusterfs.mdata=0x010000000000000000000000005dcb3a520000000011c7ae2c000000005dcb3a520000000011690f7b000000005dcb3a520000000011690f7b
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000

Also, as per 'Troubleshooting Split-Brains - Gluster Docs' there should be another 2 copies of the changelog (local + remote bricks) but in my case they are missing. Why is it like that ?

"Each file in a brick maintains the changelog of itself and that of the files present in all the other bricks in its replica set as seen by that brick."

Troubleshooting Split-Brains - Gluster Docs Project documentation for Gluster Filesystem

Thanks in advance for your response.

Best Regards,

Strahil Nikolov

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users