Re: glusterd keeps resyncing shards over and over again

Chris Drescher <info@xxxxxxxxxxxxxx> · Mon, 10 Dec 2018 13:18:10 +0100

  Hello Ravi

Thanks for your quick reply!

How can I check that?

We have about 60 shards continuously being synced. It seems those shards are always the same.

Kind regards,

Chris

On Mon, 2018-12-10 at 17:33 +0530, Ravishankar N wrote:

    On 12/10/2018 05:06 PM, Atin Mukherjee
      wrote:

      Even though the subject says the issue is with
        glusterd, I think the question is more applicable on
        heal/shards. Added the relevant folks to help out.

        On Mon, Dec 10, 2018 at 3:43 PM Chris Drescher
          <info@xxxxxxxxxxxxxx> wrote:

            Let me provide more information.

            We have 3 gluster nodes running with sharding
              activated.

            Node1: CentOS 7.5 - Glusterfs 3.12.6
            Node2: CentOS 7.5 - Glusterfs 3.12.6
            Node3: CentOS 7.5 - Glusterfs 3.12.6

            Now we updated Node 3 from CentOS 7.5 to 7.6 which
              caused a reboot.
            Glusterd Version changed from 3.12.6 to 3.12.15

            Node1: CentOS 7.5 - Glusterfs 3.12.6
            Node2: CentOS 7.5 - Glusterfs 3.12.6
            Node3: CentOS 7.6 - Glusterfs 3.12.15

            Afterwards gluster heal daemon keeps resyncing specific
              shards on bricks on Node1 and Node2. Always the same
              shards.

    Your clients (mounts)  might be experiencing disconnects from the
    brick process(es) while the same set of shards are being written to.
    Possibly to the second brick, going from the "sinks=1 " log below.
    Check if that is the case.

    -Ravi

            LOGS:

            On upgraded NODE3:

            /var/log/glusterfs/glusterd.log
            [2018-12-10 09:24:42.314624] E [MSGID: 106062]
              [glusterd-utils.c:10112:glusterd_max_opversion_use_rsp_dict]
              0-management: Maximum supported op-version not set in
              destination dictionary

            tail -f /var/log/glusterfs/glustershd.log
            [2018-12-09 04:28:05.687127] I [MSGID: 108026]
              [afr-self-heal-common.c:1726:afr_log_selfheal]
              0-data-replicate-0: Completed data selfheal on
              3f1711c2-de8c-4e8e-be10-a252f5b1b4ad. sources=[0]
              2  sinks=1 

            On NODE1:

            tail -f /var/log/glusterfs/glfsheal-data.log
            [2018-12-10 10:00:01.898139] I [MSGID: 114035]
              [client-handshake.c:202:client_set_lk_version_cbk]
              0-data-client-16: Server lk version = 1
            [2018-12-10 10:00:01.898487] I [MSGID: 114057]
              [client-handshake.c:1478:select_server_supported_programs]
              0-data-client-17: Using Program GlusterFS 3.3, Num
              (1298437), Version (330)                             
            [2018-12-10 10:00:01.898892] I [MSGID: 114046]
              [client-handshake.c:1231:client_setvolume_cbk]
              0-data-client-17: Connected to data-client-17, attached to
              remote volume '/gluster/arb2/data'.                      
            [2018-12-10 10:00:01.898900] I [MSGID: 114047]
              [client-handshake.c:1242:client_setvolume_cbk]
              0-data-client-17: Server and Client lk-version numbers are
              not same, reopening the fds                              
            [2018-12-10 10:00:01.899007] I [MSGID: 114035]
              [client-handshake.c:202:client_set_lk_version_cbk]
              0-data-client-17: Server lk version = 1
            [2018-12-10 10:00:01.901528] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-3: selecting local read_child
              data-client-9
            [2018-12-10 10:00:01.901876] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-5: selecting local read_child
              data-client-15
            [2018-12-10 10:00:01.901978] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-4: selecting local read_child
              data-client-12
            [2018-12-10 10:00:01.902708] I [MSGID: 108031]
              [afr-common.c:2376:afr_local_discovery_cbk]
              0-data-replicate-2: selecting local read_child
              data-client-6
            [2018-12-10 10:00:01.902750] I [MSGID: 104041]
              [glfs-resolve.c:971:__glfs_active_subvol] 0-data: switched
              to graph 70312d70-6f64-3031-2e6c-696e75786661 (0)

            Hope that helps!

            On Mon, 2018-12-10 at 09:22 +0100, Chris Drescher
              wrote:

              Hello everybody

              We are experiencing an urgent issue with glusterd!
              After an upgrade from centos7.5 to 7.6 our grusterfs
                keeps resyncing specific shards over and over again!

              Is this a known problem?

              This is very urgent! Please help!

              Thanks in advance!

              Kind regards.

              Chris
              _______________________________________________
              Gluster-users mailing list
              Gluster-users@xxxxxxxxxxx
              https://lists.gluster.org/mailman/listinfo/gluster-users

          _______________________________________________

          Gluster-users mailing list

          Gluster-users@xxxxxxxxxxx

          https://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users