Re: GFID Mismatch - Automatic Correction ?

Joe Julian <joe@xxxxxxxxxxxxxxxx> · Tue, 03 Jan 2017 22:32:13 -0800

Shouldn't that heal with an odd-man-out strategy? Or are all three GFIDs different?

On January 3, 2017 10:21:31 PM PST, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

      On 01/04/2017 09:31 AM, Michael Ward wrote:

        Hey,

        To give some
            more context around the initial incident.. These systems are
            hosted in AWS. The gluster brick for each instance is a
            seperate volume to the root volume. On prod-gluster01 a
            couple of nights ago we experienced massively high read iops
            on the root volume that we are unable to account for (>
            200,000 iops when it usually sits between 0 - 100 iops ).
            The box became inaccessible as a result and after
            approximately 40 minutes with no sign of the iops reducing
            was rebooted through the AWS console. 

        The GFID
            mismatch problems appeared after that. There were initially
            ~50 impacted files, but I've fixed all but 1 of them now,
            which I'm leaving broken intentionally for further testing
            if required.

        If you don't
            mind, could you have a look over the information below and
            identify anything that looks like a problem, since obviously
            we did have a bunch of GFID mismatched files, which based on
            your email shouldn't happen..

        I've included
            everything I can think of, but if there is something else
            you would like to see, please let me know.

        # gluster
            volume info gv0

        Volume Name:
            gv0

        Type: Replicate

        Volume ID:
            0ec7c49d-811c-4d4d-a3a9-e4ea9e83000c

        Status: Started

        Snapshot Count:
            0

        Number of
            Bricks: 1 x (2 + 1) = 3

        Transport-type:
            tcp

        Bricks:

        Brick1:
            prod-gluster01.fqdn.com:/export/glus_brick0/brick

        Brick2:
            prod-gluster02.fqdn.com:/export/glus_brick0/brick

        Brick3:
            prod-gluster03.fqdn.com:/export/glus_brick0/brick (arbiter)

        Options
            Reconfigured:

        cluster.favorite-child-policy:
            none

        nfs.disable: on

        performance.readdir-ahead:
            on

        client.event-threads:
            7

        server.event-threads:
            3

        performance.cache-size:
            256MB

        cluster.favorite-child-policy
            is set to none because I reverted the change to majority
            when it didn't make any difference.

        [root@prod-gluster01
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user/.viminfo

        getfattr:
            Removing leading '/' from absolute path names

        # file:
            export/glus_brick0/brick/home/user/.viminfo

        security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

        trusted.afr.dirty=0x000000000000000000000000

        trusted.bit-rot.version=0x0200000000000000585756be00024333

        trusted.gfid=0x1b86a5a76e884f40be583fa33aa9a576

        [root@prod-gluster02
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user/.viminfo

        getfattr:
            Removing leading '/' from absolute path names

        # file:
            export/glus_brick0/brick/home/user/.viminfo

        security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

        trusted.afr.dirty=0x000000000000000000000000

        trusted.afr.gv0-client-0=0x000000020000000100000000

        trusted.bit-rot.version=0x020000000000000058593aac000661fa

        trusted.gfid=0x4931b10977f34496a7cdf8f23809c372

        [root@prod-gluster03
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user/.viminfo

        getfattr:
            Removing leading '/' from absolute path names

        # file:
            export/glus_brick0/brick/home/user/.viminfo

        security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

        trusted.afr.dirty=0x000000000000000000000000

        trusted.afr.gv0-client-0=0x000000020000000100000000

        trusted.bit-rot.version=0x020000000000000058585ed6000f2077

        trusted.gfid=0x4931b10977f34496a7cdf8f23809c372

        Just in case
            it's useful, here is the getfattr for the parent directory:

        [root@prod-gluster01
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user

        getfattr:
            Removing leading '/' from absolute path names

        # file:
            export/glus_brick0/brick/home/user

        security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

        trusted.afr.dirty=0x000000000000000000000000

        trusted.afr.gv0-client-1=0x000000000000000000000000

        trusted.afr.gv0-client-2=0x000000000000000000000000

        trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193

        trusted.glusterfs.dht=0x000000010000000000000000ffffffff

        [root@prod-gluster02
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user

        getfattr:
            Removing leading '/' from absolute path names

        # file:
            export/glus_brick0/brick/home/user

        security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

        trusted.afr.dirty=0x000000000000000000000000

        trusted.afr.gv0-client-0=0x000000000000000000000000

        trusted.afr.gv0-client-2=0x000000000000000000000000

        trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193

        trusted.glusterfs.dht=0x000000010000000000000000ffffffff

        [root@prod-gluster03
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user

        getfattr:
            Removing leading '/' from absolute path names

        # file:
            export/glus_brick0/brick/home/user

        security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

        trusted.afr.dirty=0x000000000000000000000000

        trusted.afr.gv0-client-0=0x000000000000000000000000

        trusted.afr.gv0-client-1=0x000000000000000000000000

        trusted.afr.gv0-client-2=0x000000000000000000000000

        trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193

        trusted.glusterfs.dht=0x000000010000000000000000ffffffff

        [root@prod-gluster01
            bricks]# gluster volume heal gv0 info

        Brick
            prod-gluster01.fqdn.com:/export/glus_brick0/brick

        Status:
            Connected

        Number of
            entries: 0

        Brick
            prod-gluster02.fqdn.com:/export/glus_brick0/brick

        <gfid:4931b109-77f3-4496-a7cd-f8f23809c372>

        Status:
            Connected

        Number of
            entries: 1

        Brick
            prod-gluster03.fqdn.com:/export/glus_brick0/brick

        <gfid:4931b109-77f3-4496-a7cd-f8f23809c372>

        Status:
            Connected

        Number of
            entries: 1

        [root@prod-gluster01
            bricks]# gluster volume heal gv0 info split-brain

        Brick
            prod-gluster01.fqdn.com:/export/glus_brick0/brick

        Status:
            Connected

        Number of
            entries in split-brain: 0

        Brick
            prod-gluster02.fqdn.com:/export/glus_brick0/brick

        Status:
            Connected

        Number of
            entries in split-brain: 0

        Brick
            prod-gluster03.fqdn.com:/export/glus_brick0/brick

        Status:
            Connected

        Number of
            entries in split-brain: 0

        Clients show
            this in the gluster.log:

        [2017-01-04
            03:13:40.863695] W [MSGID: 108008]
            [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
            0-gv0-replicate-0: GFID mismatch for
            <gfid:0a49de7e-e4f0-4aae-9fc8-a88378e0d193>/.viminfo
            4931b109-77f3-4496-a7cd-f8f23809c372 on gv0-client-1 and
            1b86a5a7-6e88-4f40-be58-3fa33aa9a576 on gv0-client-0

        [2017-01-04
            03:13:40.867853] W [fuse-bridge.c:471:fuse_entry_cbk]
            0-glusterfs-fuse: 13067223: LOOKUP() /home/user/.viminfo
            => -1 (Input/output error)

        There's no
            mention of either of the GFID's for the .viminfo file in
            /var/log/gluster/*.log or
            /var/log/gluster/brick/export-glus_brick0-brick.log file.

    Thanks for the details Michael. While it does look like a bug, I am
    not sure how we ended in this state. Either the afr xattrs of the
    parent directory were cleared without self-heal of .vimrc  happening
    from gluster02 or 03 to 01  (or)
    it wasn't set in the first place when the file was recreated on 02
    and 03 when 01 was down. If you have some steps to re-create the
    issue, please raise a bug.

    Regards,

    Ravi

        Thank you very
            much for your time,

        Michael Ward

            From: Ravishankar N
                [mailto:ravishankar@xxxxxxxxxx]

                Sent: Wednesday, 4 January 2017 12:21 PM

                To: Michael Ward
                <Michael.Ward@xxxxxxxxxxxxxxxxxx>;
                gluster-users@xxxxxxxxxxx

                Subject: Re:  GFID Mismatch -
                Automatic Correction ?

          On 01/04/2017 06:27 AM, Michael Ward
            wrote:

          Hi,

          We have a 2 data node plus 1 arbiter node
            replicate gluster volume running gluster 3.8.5. Clients are
            also using 3.8.5.

          One of the data nodes failed the other
            night, and whilst it was down, several files were replaced
            on the second data node / arbiter (and thus the filesystem
            path was linked to a new GFID).

          When the broken node was restarted, these
            files were in a gfid mismatch state. I know how to manually
            correct them, but was wondering if there is an automated way
            ?

        For resolving
            gfid-split-brains, there is no automated way or
            favorite-child policy. When you say 2 data+1 arbiter, you
            are using an actual arbiter volume right? (as opposed to a
            replica 2 volume + a dummy node which some people are
            referring to as arbiter for server-quourm).
            gfid-split-brains should not occur on either replica-3 or
            arbiter volumes with the steps you described.

            Regards,

            Ravi

          I thought the
            cluster.favorite-child-policy volume setting of majority
            would work, but it made no difference. Clients were still
            getting Input/output error when attempting to access those
            files.

          Regards,

          Michael Ward

          _______________________________________________
          Gluster-users mailing list
          Gluster-users@xxxxxxxxxxx
          http://www.gluster.org/mailman/listinfo/gluster-users

-- 

Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users