Re: peer rejected but connected

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I've also noticed - In case it's abnormal and might help - that when rejected peer get probed it is present only on the probing peer(though as rejected). Remaining two peers do not show rejected peer in the status, and rejected peer shows only the peer it got probed from.

On 01/09/17 07:30, Gaurav Yadav wrote:
Logs from newly added node helped me in RCA of the issue.

Info file on node 10.5.6.17 consist of an additional property "tier-enabled" which is not present in info file from other 3 nodes, hence when gluster peer probe call is made, in order to maintain consistency across the cluster cksum is compared. In this case as both files are different leading to different cksum, causing state in  "State: Peer Rejected (Connected)".

This inconsistency arise due to upgrade you did.

Workaround:
1.Go to node 10.5.6.17
2.Open info file from "/var/lib/glusterd/vols/<vol-name>/info" and remove "tier-enabled=0".
3.Restart glusterd services
4.Peer probe again.

Thanks
Gaurav

On Thu, Aug 31, 2017 at 3:37 PM, lejeczek <peljasz@xxxxxxxxxxx <mailto:peljasz@xxxxxxxxxxx>> wrote:

    attached the lot as per your request.

    Would bee really great if you can find the root cause
    of this and suggest a resolution. Fingers crossed.
    thanks, L.

    On 31/08/17 05:34, Gaurav Yadav wrote:

        Could you please sendentire content of
        "/var/lib/glusterd/" directory of the 4th node
        which is being peer probed, along with
        command-history and glusterd.logs.

        Thanks
        Gaurav

        On Wed, Aug 30, 2017 at 7:10 PM, lejeczek
        <peljasz@xxxxxxxxxxx <mailto:peljasz@xxxxxxxxxxx>
        <mailto:peljasz@xxxxxxxxxxx
        <mailto:peljasz@xxxxxxxxxxx>>> wrote:



            On 30/08/17 07:18, Gaurav Yadav wrote:


                Could you please send me "info" file which is
                placed in "/var/lib/glusterd/vols/<vol-name>"
                directory from all the nodes along with
                glusterd.logs and command-history.

                Thanks
                Gaurav

                On Tue, Aug 29, 2017 at 7:13 PM, lejeczek
                <peljasz@xxxxxxxxxxx
        <mailto:peljasz@xxxxxxxxxxx>
        <mailto:peljasz@xxxxxxxxxxx
        <mailto:peljasz@xxxxxxxxxxx>>
                <mailto:peljasz@xxxxxxxxxxx
        <mailto:peljasz@xxxxxxxxxxx>

                <mailto:peljasz@xxxxxxxxxxx
        <mailto:peljasz@xxxxxxxxxxx>>>> wrote:

                    hi fellas,
                    same old same
                    in log of the probing peer I see:
                    ...
                    2017-08-29 13:36:16.882196] I [MSGID:
        106493]

               
        [glusterd-handler.c:3020:__glusterd_handle_probe_query]
                    0-glusterd: Responded to
        priv.xx.xx.priv.xx.xx.x,
                    op_ret: 0, op_errno: 0, ret: 0
                    [2017-08-29 13:36:16.904961] I [MSGID:
        106490]

               
        [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req]
                    0-glusterd: Received probe from uuid:
                    2a17edb4-ae68-4b67-916e-e38a2087ca28
                    [2017-08-29 13:36:16.906477] E [MSGID:
        106010]

               
        [glusterd-utils.c:3034:glusterd_compare_friend_volume]
                    0-management: Version of Cksums CO-DATA
                differ. local
                    cksum = 4088157353, remote cksum =
        2870780063
                on peer
                    10.5.6.17
                    [2017-08-29 13:36:16.907187] I [MSGID:
        106493]

               
        [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp]
                    0-glusterd: Responded to 10.5.6.17
        (0), ret:
                0, op_ret: -1
                    ...

                    Why would adding a new peer make
        cluster jump
                to check
                    checksums on a vol on that newly added
        peer?


            really. I mean, no brick even exists on newly
        added
            peer, it's just been probed, why this?:

            [2017-08-30 13:17:51.949430] E [MSGID: 106010]
           
        [glusterd-utils.c:3034:glusterd_compare_friend_volume]
            0-management: Version of Cksums CO-DATA
        differ. local
            cksum = 4088157353, remote cksum = 2870780063
        on peer
            10.5.6.17

            10.5.6.17 is a candidate I'm probing from a
        working
            cluster.
            Why gluster wants checksums and why checksums
        would be
            different?
            Would anybody know what is going on there?


                    Is it why the peer gets rejected?
                    That peer I'm hoping to add, was a
        member of the
                    cluster in the past but I did "usual"
        wipe of
                    /var/lib/gluster on candidate peer.

                    a hint, solution would be great to hear.
                    L.
                   
        _______________________________________________
                    Gluster-users mailing list
        Gluster-users@xxxxxxxxxxx
        <mailto:Gluster-users@xxxxxxxxxxx>
                <mailto:Gluster-users@xxxxxxxxxxx
        <mailto:Gluster-users@xxxxxxxxxxx>>
                    <mailto:Gluster-users@xxxxxxxxxxx
        <mailto:Gluster-users@xxxxxxxxxxx>
                <mailto:Gluster-users@xxxxxxxxxxx
        <mailto:Gluster-users@xxxxxxxxxxx>>>
        http://lists.gluster.org/mailman/listinfo/gluster-users
        <http://lists.gluster.org/mailman/listinfo/gluster-users>
               
        <http://lists.gluster.org/mailman/listinfo/gluster-users
        <http://lists.gluster.org/mailman/listinfo/gluster-users>>

               
        <http://lists.gluster.org/mailman/listinfo/gluster-users
        <http://lists.gluster.org/mailman/listinfo/gluster-users>
               
        <http://lists.gluster.org/mailman/listinfo/gluster-users
        <http://lists.gluster.org/mailman/listinfo/gluster-users>>>



            _______________________________________________
            Gluster-users mailing list
        Gluster-users@xxxxxxxxxxx
        <mailto:Gluster-users@xxxxxxxxxxx>
            <mailto:Gluster-users@xxxxxxxxxxx
        <mailto:Gluster-users@xxxxxxxxxxx>>
        http://lists.gluster.org/mailman/listinfo/gluster-users
        <http://lists.gluster.org/mailman/listinfo/gluster-users>
           
        <http://lists.gluster.org/mailman/listinfo/gluster-users
        <http://lists.gluster.org/mailman/listinfo/gluster-users>>





_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux