Re: RE : Frequent connect and disconnect messages flooded in logs

Mohammed Rafi K C <rkavunga@xxxxxxxxxx> · Wed, 7 Dec 2016 22:38:12 +0530

    Hi Micha,
    This is great. I will provide you one debug build which has two
      fixes which I possible suspect for a frequent disconnect issue,
      though I don't have much data to validate my theory. So I will
      take one more day to dig in to that.
    Thanks for your support, and opensource++  
    Regards
    Rafi KC

    On 12/07/2016 05:02 AM, Micha Ober
      wrote:

      Hi,

        thank you for your answer and even more for the question!

        Until now, I was using FUSE. Today I changed all mounts to NFS
        using the same 3.7.17 version.

        But: The problem is still the same. Now, the NFS logfile
        contains lines like these:

        [2016-12-06 15:12:29.006325] C
        [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired]
        0-gv0-client-7: server X.X.18.62:49153 has not responded in the
        last 42 seconds, disconnecting.

        Interestingly enough,  the IP address X.X.18.62 is the same
        machine! As I wrote earlier, each node serves both as a server
        and a client, as each node contributes bricks to the volume.
        Every server is connecting to itself via its hostname. For
        example, the fstab on the node "giant2" looks like:

        #giant2:/gv0    /shared_data    glusterfs       defaults,noauto
        0       0

        #giant2:/gv2    /shared_slurm   glusterfs       defaults,noauto
        0       0

        giant2:/gv0     /shared_data    nfs            
        defaults,_netdev,vers=3 0       0

        giant2:/gv2     /shared_slurm   nfs            
        defaults,_netdev,vers=3 0       0

        So I understand the disconnects even less. 

        I don't know if it's possible to create a dummy cluster which
        exposes the same behaviour, because the disconnects only happen
        when there are compute jobs running on those nodes - and they
        are GPU compute jobs, so that's something which cannot be easily
        emulated in a VM.

        As we have more clusters (which are running fine with an ancient
        3.4 version :-)) and we are currently not dependent on this
        particular cluster (which may stay like this for this month, I
        think) I should be able to deploy the debug build on the "real"
        cluster, if you can provide a debug build.

        Regards and thanks,

        Micha

        Am 06.12.2016 um 08:15 schrieb Mohammed Rafi K C:

        On 12/03/2016 12:56 AM, Micha Ober
          wrote:

          ** Update: ** I have
              downgraded from 3.8.6 to 3.7.17 now, but the problem still
              exists.

            Client log: http://paste.ubuntu.com/23569065/

            Brick log: http://paste.ubuntu.com/23569067/

            Please note that each server has two bricks.

            Whereas, according to the logs, one brick loses the
              connection to all other hosts:

            [2016-12-02 18:38:53.703301] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.219:49121 failed (Broken pipe)
[2016-12-02 18:38:53.703381] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.62:49118 failed (Broken pipe)
[2016-12-02 18:38:53.703380] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.107:49121 failed (Broken pipe)
[2016-12-02 18:38:53.703424] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.206:49120 failed (Broken pipe)
[2016-12-02 18:38:53.703359] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.58:49121 failed (Broken pipe)

The SECOND brick on the SAME host is NOT affected, i.e. no disconnects!
As I said, the network connection is fine and the disks are idle.
The CPU always has 2 free cores.

It looks like I have to downgrade to 3.4 now in order for the disconnects to stop.

        Hi Micha,

        Thanks for the update and sorry for what happened with gluster
        higher versions. I can understand the need for downgrade as it
        is a production setup.

        Can you tell me the clients used here ? whether it is a
        fuse,nfs,nfs-ganesha, smb or libgfapi ?

        Since I'm not able to reproduce the issue (I have been trying
        from last 3days) and the logs are not much helpful here (we
        don't have much logs in socket layer), Could you please create a
        dummy cluster and try to reproduce the issue? If then we can
        play with that volume and I could provide some debug build which
        we can use for further debugging?

        If you don't have bandwidth for this, please leave it ;).

        Regards

        Rafi KC

            - Micha

            Am 30.11.2016 um 06:57 schrieb Mohammed Rafi K C:

            Hi Micha,
            I have changed the thread and subject so that your
              original thread remain same for your query. Let's try to
              fix the problem what you observed with 3.8.4, So I have
              started a new thread to discuss the frequent disconnect
              problem.
            If any one else has experienced the same problem,
                please respond to the mail.

            It would be very helpful if you could give us some more
              logs from clients and bricks.  Also any reproducible steps
              will surely help to chase the problem further.
            Regards
            Rafi KC

            On 11/30/2016 04:44 AM, Micha
              Ober wrote:

                  I had opened another thread on this
                      mailing list (Subject: "After upgrade from 3.4.2
                      to 3.8.5 - High CPU usage resulting in disconnects
                      and split-brain").

                  The title may be a bit misleading now,
                      as I am no longer observing high CPU usage after
                      upgrading to 3.8.6, but the disconnects are still
                      happening and the number of files in split-brain
                      is growing.

                  Setup: 6 compute nodes, each serving as
                      a glusterfs server and client, Ubuntu 14.04, two
                      bricks per node, distribute-replicate

                  I have two gluster volumes set up (one
                      for scratch data, one for the slurm scheduler).
                      Only the scratch data volume shows critical errors
                      "[...] has not responded in the last 42 seconds,
                      disconnecting.". So I can rule out network
                      problems, the gigabit link between the nodes is
                      not saturated at all. The disks are almost idle
                      (<10%).

                  I have glusterfs 3.4.2 on Ubuntu 12.04
                      on a another compute cluster, running fine since
                      it was deployed.
                  I had glusterfs 3.4.2 on Ubuntu 14.04
                      on this cluster, running fine for almost a year.

                  After upgrading to 3.8.5, the problems
                      (as described) started. I would like to use some
                      of the new features of the newer versions (like
                      bitrot), but the users can't run their compute
                      jobs right now because the result files are
                      garbled.

                  There also seems to be a bug report
                      with a smiliar problem: (but no progress)
                  https://bugzilla.redhat.com/show_bug.cgi?id=1370683

                  For me, ALL servers are affected (not
                      isolated to one or two servers)

                  I also see messages like "INFO:
                        task gpu_graphene_bv:4476 blocked for more than
                        120 seconds." in the syslog.

                  For completeness (gv0 is the scratch
                      volume, gv2 the slurm volume):

                  [root@giant2: ~]# gluster v info

                  Volume Name: gv0
                  Type: Distributed-Replicate
                  Volume ID:
                      993ec7c9-e4bc-44d0-b7c4-2d977e622e86
                  Status: Started
                  Snapshot Count: 0
                  Number of Bricks: 6 x 2 = 12
                  Transport-type: tcp
                  Bricks:
                  Brick1: giant1:/gluster/sdc/gv0
                  Brick2: giant2:/gluster/sdc/gv0
                  Brick3: giant3:/gluster/sdc/gv0
                  Brick4: giant4:/gluster/sdc/gv0
                  Brick5: giant5:/gluster/sdc/gv0
                  Brick6: giant6:/gluster/sdc/gv0
                  Brick7: giant1:/gluster/sdd/gv0
                  Brick8: giant2:/gluster/sdd/gv0
                  Brick9: giant3:/gluster/sdd/gv0
                  Brick10: giant4:/gluster/sdd/gv0
                  Brick11: giant5:/gluster/sdd/gv0
                  Brick12: giant6:/gluster/sdd/gv0
                  Options Reconfigured:
                  auth.allow: X.X.X.*,127.0.0.1
                  nfs.disable: on

                  Volume Name: gv2
                  Type: Replicate
                  Volume ID:
                      30c78928-5f2c-4671-becc-8deaee1a7a8d
                  Status: Started
                  Snapshot Count: 0
                  Number of Bricks: 1 x 2 = 2
                  Transport-type: tcp
                  Bricks:
                  Brick1: giant1:/gluster/sdd/gv2
                  Brick2: giant2:/gluster/sdd/gv2
                  Options Reconfigured:
                  auth.allow: X.X.X.*,127.0.0.1
                  cluster.granular-entry-heal: on
                  cluster.locking-scheme: granular
                  nfs.disable: on

                2016-11-30 0:10 GMT+01:00 Micha
                  Ober <micha2k@xxxxxxxxx>:

                      There
                        also seems to be a bug report with a smiliar
                        problem: (but no progress)
                      https://bugzilla.redhat.com/show_bug.cgi?id=1370683

                      For me, ALL servers are affected
                          (not isolated to one or two servers)

                      I also see messages like "INFO:
                            task gpu_graphene_bv:4476 blocked for more
                            than 120 seconds." in the syslog.

                      For completeness (gv0 is the
                          scratch volume, gv2 the slurm volume):

                          [root@giant2: ~]#
                            gluster v info

                          Volume Name: gv0
                          Type:
                            Distributed-Replicate
                          Volume ID:
                            993ec7c9-e4bc-44d0-b7c4-2d977e622e86
                          Status: Started
                          Snapshot Count: 0
                          Number of Bricks: 6
                            x 2 = 12
                          Transport-type: tcp
                          Bricks:
                          Brick1:
                            giant1:/gluster/sdc/gv0
                          Brick2:
                            giant2:/gluster/sdc/gv0
                          Brick3:
                            giant3:/gluster/sdc/gv0
                          Brick4:
                            giant4:/gluster/sdc/gv0
                          Brick5:
                            giant5:/gluster/sdc/gv0
                          Brick6:
                            giant6:/gluster/sdc/gv0
                          Brick7:
                            giant1:/gluster/sdd/gv0
                          Brick8:
                            giant2:/gluster/sdd/gv0
                          Brick9:
                            giant3:/gluster/sdd/gv0
                          Brick10:
                            giant4:/gluster/sdd/gv0
                          Brick11:
                            giant5:/gluster/sdd/gv0
                          Brick12:
                            giant6:/gluster/sdd/gv0
                          Options
                            Reconfigured:
                          auth.allow:
                            X.X.X.*,127.0.0.1
                          nfs.disable: on

                          Volume Name: gv2
                          Type: Replicate
                          Volume ID:
                            30c78928-5f2c-4671-becc-8deaee1a7a8d
                          Status: Started
                          Snapshot Count: 0
                          Number of Bricks: 1
                            x 2 = 2
                          Transport-type: tcp
                          Bricks:
                          Brick1:
                            giant1:/gluster/sdd/gv2
                          Brick2:
                            giant2:/gluster/sdd/gv2
                          Options
                            Reconfigured:
                          auth.allow:
                            X.X.X.*,127.0.0.1
                          cluster.granular-entry-heal:
                            on
                          cluster.locking-scheme:
                            granular
                          nfs.disable: on

                          2016-11-29 19:21
                            GMT+01:00 Micha Ober <micha2k@xxxxxxxxx>:

                                I
                                  had opened another thread on this
                                  mailing list (Subject: "After upgrade
                                  from 3.4.2 to 3.8.5 - High CPU usage
                                  resulting in disconnects and
                                  split-brain").

                                The
                                  title may be a bit misleading now, as
                                  I am no longer observing high CPU
                                  usage after upgrading to 3.8.6, but
                                  the disconnects are still happening
                                  and the number of files in split-brain
                                  is growing.

                                Setup:
                                  6 compute nodes, each serving as a
                                  glusterfs server and client, Ubuntu
                                  14.04, two bricks per node,
                                  distribute-replicate

                                I
                                  have two gluster volumes set up (one
                                  for scratch data, one for the slurm
                                  scheduler). Only the scratch data
                                  volume shows critical errors "[...]
                                  has not responded in the last 42
                                  seconds, disconnecting.". So I can
                                  rule out network problems, the gigabit
                                  link between the nodes is not
                                  saturated at all. The disks are almost
                                  idle (<10%).

                                I
                                  have glusterfs 3.4.2 on Ubuntu 12.04
                                  on a another compute cluster, running
                                  fine since it was deployed.
                                I
                                  had glusterfs 3.4.2 on Ubuntu 14.04 on
                                  this cluster, running fine for almost
                                  a year.

                                After
                                  upgrading to 3.8.5, the problems (as
                                  described) started. I would like to
                                  use some of the new features of the
                                  newer versions (like bitrot), but the
                                  users can't run their compute jobs
                                  right now because the result files are
                                  garbled.

                                    2016-11-29
                                      18:53 GMT+01:00 Atin Mukherjee <amukherj@xxxxxxxxxx>:

                                        Would you be able to share what is not working for you in 3.8.x (mention the exact version). 3.4 is quite old and falling back to an unsupported version doesn't look a feasible option.

                                              On Tue, 29
                                                Nov 2016 at 17:01, Micha
                                                Ober <micha2k@xxxxxxxxx>
                                                wrote:

                                                  Hi,

                                                  I was using gluster 3.4 and
                                                    upgraded to 3.8, but
                                                    that version showed
                                                    to be unusable for
                                                    me. I now need to
                                                    downgrade.

                                                  I'm running Ubuntu 14.04. As
                                                    upgrades of the op
                                                    version
                                                    are irreversible, I
                                                    guess I have to
                                                    delete all gluster
                                                    volumes and
                                                    re-create them with
                                                    the downgraded
                                                    version. 

                                                  0. Backup data
                                                  1. Unmount all gluster volumes
                                                  2. apt-get purge
                                                    glusterfs-server
                                                    glusterfs-client
                                                  3. Remove PPA for 3.8
                                                  4. Add PPA for older version
                                                  5. apt-get install
                                                    glusterfs-server
                                                    glusterfs-client
                                                  6. Create volumes

                                                  Is "purge" enough to delete all
                                                    configuration files
                                                    of the currently
                                                    installed version or
                                                    do I need to
                                                     manually clear some
                                                    residues before
                                                    installing an older
                                                    version?

                                                  Thanks.

                                              _______________________________________________

                                              Gluster-users mailing list

                                              Gluster-users@xxxxxxxxxxx

                                              http://www.gluster.org/mailman/listinfo/gluster-users

                                            -- 

                                            -
                                              Atin (atinm)

              _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users