Re: RE : Frequent connect and disconnect messages flooded in logs

Micha Ober <micha2k@xxxxxxxxx> · Mon, 19 Dec 2016 19:05:57 +0100

    Hi,

      the log does not show anything like that. Also, I'm using ext4 on
      the bricks.

      The log only contains entries like these:

      [Fri Nov 25 14:23:27 2016] INFO: task gpu_graphene_bv:4476 blocked
      for more than 120 seconds.

      [Fri Nov 25 14:23:27 2016]       Tainted: P           OE 
      3.19.0-25-generic #26~14.04.1-Ubuntu

      [Fri Nov 25 14:23:27 2016] "echo 0 >
      /proc/sys/kernel/hung_task_timeout_secs" disables this message.

      [Fri Nov 25 14:23:27 2016] gpu_graphene_bv D ffff8804aa39be08    
      0  4476   4461 0x00000000

      [Fri Nov 25 14:23:27 2016]  ffff8804aa39be08 ffff8804ad0febf0
      0000000000013e80 ffff8804aa39bfd8

      [Fri Nov 25 14:23:27 2016]  0000000000013e80 ffff8804ad403110
      ffff8804ad0febf0 ffff8804aa39be18

      [Fri Nov 25 14:23:27 2016]  ffff8804aa2c87d0 ffff88049df2e000
      ffff8804aa39be30 ffff8804aa2c88a0

      [Fri Nov 25 14:23:27 2016] Call Trace:

      [Fri Nov 25 14:23:27 2016]  [<ffffffff817b22e9>]
      schedule+0x29/0x70

      [Fri Nov 25 14:23:27 2016]  [<ffffffff812dc06d>]
      __fuse_request_send+0x11d/0x290

      [Fri Nov 25 14:23:27 2016]  [<ffffffff810b4e10>] ?
      prepare_to_wait_event+0x110/0x110

      [Fri Nov 25 14:23:27 2016]  [<ffffffff812dc1f2>]
      fuse_request_send+0x12/0x20

      [Fri Nov 25 14:23:27 2016]  [<ffffffff812e576d>]
      fuse_flush+0x12d/0x180

      [Fri Nov 25 14:23:27 2016]  [<ffffffff811e9973>]
      filp_close+0x33/0x80

      [Fri Nov 25 14:23:27 2016]  [<ffffffff8120a152>]
      __close_fd+0x82/0xa0

      [Fri Nov 25 14:23:27 2016]  [<ffffffff811e99e3>]
      SyS_close+0x23/0x50

      [Fri Nov 25 14:23:27 2016]  [<ffffffff817b668d>]
      system_call_fastpath+0x16/0x1b

      Which is due to the file system not responding, I guess.

      Since I switched the mounts from FUSE to NFS, occasionally I also
      see:

      [Wed Dec 14 23:42:47 2016] nfs: server giant2 not responding,
      still trying

      [Wed Dec 14 23:43:12 2016] nfs: server giant2 not responding,
      still trying

      [Wed Dec 14 23:45:04 2016] nfs: server giant2 OK

      [Wed Dec 14 23:45:04 2016] nfs: server giant2 OK

      In another post you asked for logfiles with TRACE loglevel, I'll
      provide them shortly.

      Best regards and thanks,

      Micha

      Am 19.12.2016 um 16:09 schrieb Mohammed Rafi K C:

      Hi Micha,
      Can you please also see if there is any error messages in dmesg
        ? Basically I'm trying to see whether your hitting issues
        described in https://bugzilla.kernel.org/show_bug.cgi?id=73831
        .

      Regards
      Rafi KC

      On 12/19/2016 11:58 AM, Mohammed Rafi
        K C wrote:

        Hi Micha,
        Sorry for the late reply. I was busy with some other things.
        If you have still the setup available Can you enable TRACE
          log level [1],[2] and see if you could find any log entries
          when the network start disconnecting. Basically I'm trying to
          find out any disconnection had occurred other than ping timer
          expire issue.

        [1] : gluster volume <volname>
          diagnostics.brick-log-level TRACE
        [2] : gluster volume <volname>
          diagnostics.client-log-level TRACE

        Regards
        Rafi KC

        On 12/08/2016 07:59 PM, Atin
          Mukherjee wrote:

              On Thu, Dec 8, 2016 at 4:37 PM,
                Micha Ober <micha2k@xxxxxxxxx>
                wrote:

                    Hi
                      Rafi,

                      thank you for your support. It is greatly
                      appreciated.

                      Just some more thoughts from my side:

                      There have been no reports from other  users in
                      *this* thread until now, but I have found at least
                      one user with a very simiar problem in an older
                      thread:

                      https://www.gluster.org/pipermail/gluster-users/2014-November/019637.html

                      He is also reporting disconnects  with no apparent
                      reasons, althogh his setup is a bit more
                      complicated, also involving a firewall. In our
                      setup, all servers/clients are connected via 1 GbE
                      with no firewall or anything that might
                      block/throttle traffic. Also, we are using exactly
                      the same software versions on all nodes.

                      I can also find some reports in the bugtracker
                      when searching for "rpc_client_ping_timer_expired"
                      and "rpc_clnt_ping_timer_expired" (looks like
                      spelling changed during versions).

                      https://bugzilla.redhat.com/show_bug.cgi?id=1096729

                Just FYI, this is a different issue, here GlusterD
                  fails to handle the volume of incoming requests on
                  time since MT-epoll is not enabled here.

                      https://bugzilla.redhat.com/show_bug.cgi?id=1370683

                      But both reports involve large traffic/load on the
                      bricks/disks, which is not the case for out setup.

                      To give a ballpark figure: Over three days, 30 GiB
                      were written. And the data was not written at
                      once, but continuously over the whole time.

                      Just to be sure, I have checked the logfiles of
                      one of the other clusters right now, which are
                      sitting in the same building, in the same rack,
                      even on the same switch, running the same jobs,
                      but with glusterfs 3.4.2 and I can see no
                      disconnects in the logfiles. So I can definitely
                      rule out our infrastructure as problem.

                      Regards,

                      Micha

                          Am 07.12.2016 um 18:08 schrieb Mohammed Rafi K
                          C:

                          Hi Micha,
                          This is great. I will provide you one debug
                            build which has two fixes which I possible
                            suspect for a frequent disconnect issue,
                            though I don't have much data to validate my
                            theory. So I will take one more day to dig
                            in to that.
                          Thanks for your support, and opensource++ 

                          Regards
                          Rafi KC

                          On
                            12/07/2016 05:02 AM, Micha Ober wrote:

                            Hi,

                              thank you for your answer and even more
                              for the question!

                              Until now, I was using FUSE. Today I
                              changed all mounts to NFS using the same
                              3.7.17 version.

                              But: The problem is still the same. Now,
                              the NFS logfile contains lines like these:

                              [2016-12-06 15:12:29.006325] C
                              [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired]
                              0-gv0-client-7: server X.X.18.62:49153 has
                              not responded in the last 42 seconds,
                              disconnecting.

                              Interestingly enough,  the IP address
                              X.X.18.62 is the same machine! As I wrote
                              earlier, each node serves both as a server
                              and a client, as each node contributes
                              bricks to the volume. Every server is
                              connecting to itself via its hostname. For
                              example, the fstab on the node "giant2"
                              looks like:

                              #giant2:/gv0    /shared_data   
                              glusterfs       defaults,noauto 0       0

                              #giant2:/gv2    /shared_slurm  
                              glusterfs       defaults,noauto 0       0

                              giant2:/gv0     /shared_data   
                              nfs             defaults,_netdev,vers=3
                              0       0

                              giant2:/gv2     /shared_slurm  
                              nfs             defaults,_netdev,vers=3
                              0       0

                              So I understand the disconnects even less.

                              I don't know if it's possible to create a
                              dummy cluster which exposes the same
                              behaviour, because the disconnects only
                              happen when there are compute jobs running
                              on those nodes - and they are GPU compute
                              jobs, so that's something which cannot be
                              easily emulated in a VM.

                              As we have more clusters (which are
                              running fine with an ancient 3.4 version
                              :-)) and we are currently not dependent on
                              this particular cluster (which may stay
                              like this for this month, I think) I
                              should be able to deploy the debug build
                              on the "real" cluster, if you can provide
                              a debug build.

                              Regards and thanks,

                              Micha

                              Am 06.12.2016 um 08:15 schrieb Mohammed
                              Rafi K C:

                              On
                                12/03/2016 12:56 AM, Micha Ober wrote:

                                **
                                    Update: ** I have downgraded from
                                    3.8.6 to 3.7.17 now, but the problem
                                    still exists.

                                  Client log: http://paste.ubuntu.com/23569065/

                                  Brick log: http://paste.ubuntu.com/23569067/

                                  Please note that each server
                                    has two bricks.

                                  Whereas, according to the
                                    logs, one brick loses the connection
                                    to all other hosts:

                                  [2016-12-02 18:38:53.703301] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.219:49121 failed (Broken pipe)
[2016-12-02 18:38:53.703381] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.62:49118 failed (Broken pipe)
[2016-12-02 18:38:53.703380] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.107:49121 failed (Broken pipe)
[2016-12-02 18:38:53.703424] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.206:49120 failed (Broken pipe)
[2016-12-02 18:38:53.703359] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.58:49121 failed (Broken pipe)

The SECOND brick on the SAME host is NOT affected, i.e. no disconnects!
As I said, the network connection is fine and the disks are idle.
The CPU always has 2 free cores.

It looks like I have to downgrade to 3.4 now in order for the disconnects to stop.

                              Hi Micha,

                              Thanks for the update and sorry for what
                              happened with gluster higher versions. I
                              can understand the need for downgrade as
                              it is a production setup.

                              Can you tell me the clients used here ?
                              whether it is a fuse,nfs,nfs-ganesha, smb
                              or libgfapi ?

                              Since I'm not able to reproduce the issue
                              (I have been trying from last 3days) and
                              the logs are not much helpful here (we
                              don't have much logs in socket layer),
                              Could you please create a dummy cluster
                              and try to reproduce the issue? If then we
                              can play with that volume and I could
                              provide some debug build which we can use
                              for further debugging?

                              If you don't have bandwidth for this,
                              please leave it ;).

                              Regards

                              Rafi KC

                                  - Micha

                                  Am 30.11.2016 um 06:57 schrieb
                                  Mohammed Rafi K C:

                                  Hi Micha,
                                  I have changed the thread and
                                    subject so that your original thread
                                    remain same for your query. Let's
                                    try to fix the problem what you
                                    observed with 3.8.4, So I have
                                    started a new thread to discuss the
                                    frequent disconnect problem.
                                  If any one else has experienced
                                      the same problem, please respond
                                      to the mail.

                                  It would be very helpful if you
                                    could give us some more logs from
                                    clients and bricks.  Also any
                                    reproducible steps will surely help
                                    to chase the problem further.
                                  Regards
                                  Rafi KC

                                  On
                                    11/30/2016 04:44 AM, Micha Ober
                                    wrote:

                                        I had opened
                                            another thread on this
                                            mailing list (Subject:
                                            "After upgrade from 3.4.2 to
                                            3.8.5 - High CPU usage
                                            resulting in disconnects and
                                            split-brain").

                                        The title may be
                                            a bit misleading now, as I
                                            am no longer observing high
                                            CPU usage after upgrading to
                                            3.8.6, but the disconnects
                                            are still happening and the
                                            number of files in
                                            split-brain is growing.

                                        Setup: 6 compute
                                            nodes, each serving as a
                                            glusterfs server and client,
                                            Ubuntu 14.04, two bricks per
                                            node, distribute-replicate

                                        I have two
                                            gluster volumes set up (one
                                            for scratch data, one for
                                            the slurm scheduler). Only
                                            the scratch data volume
                                            shows critical errors "[...]
                                            has not responded in the
                                            last 42 seconds,
                                            disconnecting.". So I can
                                            rule out network problems,
                                            the gigabit link between the
                                            nodes is not saturated at
                                            all. The disks are almost
                                            idle (<10%).

                                        I have glusterfs
                                            3.4.2 on Ubuntu 12.04 on a
                                            another compute cluster,
                                            running fine since it was
                                            deployed.
                                        I had glusterfs
                                            3.4.2 on Ubuntu 14.04 on
                                            this cluster, running fine
                                            for almost a year.

                                        After upgrading
                                            to 3.8.5, the problems (as
                                            described) started. I would
                                            like to use some of the new
                                            features of the newer
                                            versions (like bitrot), but
                                            the users can't run their
                                            compute jobs right now
                                            because the result files are
                                            garbled.

                                        There also seems
                                            to be a bug report with a
                                            smiliar problem: (but no
                                            progress)
                                        https://bugzilla.redhat.com/show_bug.cgi?id=1370683

                                        For me, ALL
                                            servers are affected (not
                                            isolated to one or two
                                            servers)

                                        I also see
                                            messages like "INFO:
                                              task gpu_graphene_bv:4476
                                              blocked for more than 120
                                              seconds." in the
                                            syslog.

                                        For completeness
                                            (gv0 is the scratch volume,
                                            gv2 the slurm volume):

                                        [root@giant2: ~]#
                                            gluster v info

                                        Volume Name: gv0
                                        Type:
                                            Distributed-Replicate
                                        Volume ID:
                                            993ec7c9-e4bc-44d0-b7c4-2d977e622e86
                                        Status: Started
                                        Snapshot Count: 0
                                        Number of Bricks:
                                            6 x 2 = 12
                                        Transport-type:
                                            tcp
                                        Bricks:
                                        Brick1:
                                            giant1:/gluster/sdc/gv0
                                        Brick2:
                                            giant2:/gluster/sdc/gv0
                                        Brick3:
                                            giant3:/gluster/sdc/gv0
                                        Brick4:
                                            giant4:/gluster/sdc/gv0
                                        Brick5:
                                            giant5:/gluster/sdc/gv0
                                        Brick6:
                                            giant6:/gluster/sdc/gv0
                                        Brick7:
                                            giant1:/gluster/sdd/gv0
                                        Brick8:
                                            giant2:/gluster/sdd/gv0
                                        Brick9:
                                            giant3:/gluster/sdd/gv0
                                        Brick10:
                                            giant4:/gluster/sdd/gv0
                                        Brick11:
                                            giant5:/gluster/sdd/gv0
                                        Brick12:
                                            giant6:/gluster/sdd/gv0
                                        Options
                                            Reconfigured:
                                        auth.allow:
                                            X.X.X.*,127.0.0.1
                                        nfs.disable: on

                                        Volume Name: gv2
                                        Type: Replicate
                                        Volume ID:
                                            30c78928-5f2c-4671-becc-8deaee1a7a8d
                                        Status: Started
                                        Snapshot Count: 0
                                        Number of Bricks:
                                            1 x 2 = 2
                                        Transport-type:
                                            tcp
                                        Bricks:
                                        Brick1:
                                            giant1:/gluster/sdd/gv2
                                        Brick2:
                                            giant2:/gluster/sdd/gv2
                                        Options
                                            Reconfigured:
                                        auth.allow:
                                            X.X.X.*,127.0.0.1
                                        cluster.granular-entry-heal:
                                            on
                                        cluster.locking-scheme:
                                            granular
                                        nfs.disable: on

                                      2016-11-30
                                        0:10 GMT+01:00 Micha Ober <micha2k@xxxxxxxxx>:

                                            There
                                              also seems to be a bug
                                              report with a smiliar
                                              problem: (but no progress)
                                            https://bugzilla.redhat.com/show_bug.cgi?id=1370683

                                            For me, ALL
                                                servers are affected
                                                (not isolated to one or
                                                two servers)

                                            I also see
                                                messages like "INFO:
                                                  task
                                                  gpu_graphene_bv:4476
                                                  blocked for more than
                                                  120 seconds." in
                                                the syslog.

                                            For
                                                completeness (gv0 is the
                                                scratch volume, gv2 the
                                                slurm volume):

                                                [root@giant2: ~]#
                                                  gluster v info

                                                Volume Name: gv0
                                                Type:
                                                  Distributed-Replicate
                                                Volume ID:
                                                  993ec7c9-e4bc-44d0-b7c4-2d977e622e86
                                                Status: Started
                                                Snapshot Count: 0
                                                Number of Bricks: 6
                                                  x 2 = 12
                                                Transport-type: tcp
                                                Bricks:
                                                Brick1:
                                                  giant1:/gluster/sdc/gv0
                                                Brick2:
                                                  giant2:/gluster/sdc/gv0
                                                Brick3:
                                                  giant3:/gluster/sdc/gv0
                                                Brick4:
                                                  giant4:/gluster/sdc/gv0
                                                Brick5:
                                                  giant5:/gluster/sdc/gv0
                                                Brick6:
                                                  giant6:/gluster/sdc/gv0
                                                Brick7:
                                                  giant1:/gluster/sdd/gv0
                                                Brick8:
                                                  giant2:/gluster/sdd/gv0
                                                Brick9:
                                                  giant3:/gluster/sdd/gv0
                                                Brick10:
                                                  giant4:/gluster/sdd/gv0
                                                Brick11:
                                                  giant5:/gluster/sdd/gv0
                                                Brick12:
                                                  giant6:/gluster/sdd/gv0
                                                Options
                                                  Reconfigured:
                                                auth.allow:
                                                  X.X.X.*,127.0.0.1
                                                nfs.disable: on

                                                Volume Name: gv2
                                                Type: Replicate
                                                Volume ID:
                                                  30c78928-5f2c-4671-becc-8deaee1a7a8d
                                                Status: Started
                                                Snapshot Count: 0
                                                Number of Bricks: 1
                                                  x 2 = 2
                                                Transport-type: tcp
                                                Bricks:
                                                Brick1:
                                                  giant1:/gluster/sdd/gv2
                                                Brick2:
                                                  giant2:/gluster/sdd/gv2
                                                Options
                                                  Reconfigured:
                                                auth.allow:
                                                  X.X.X.*,127.0.0.1
                                                cluster.granular-entry-heal:
                                                  on
                                                cluster.locking-scheme:
                                                  granular
                                                nfs.disable: on

                                                2016-11-29
                                                  19:21 GMT+01:00 Micha
                                                  Ober <micha2k@xxxxxxxxx>:

                                                      I
                                                        had opened
                                                        another thread
                                                        on this mailing
                                                        list (Subject:
                                                        "After upgrade
                                                        from 3.4.2 to
                                                        3.8.5 - High CPU
                                                        usage resulting
                                                        in disconnects
                                                        and
                                                        split-brain").

                                                      The
                                                        title may be a
                                                        bit misleading
                                                        now, as I am no
                                                        longer observing
                                                        high CPU usage
                                                        after upgrading
                                                        to 3.8.6, but
                                                        the disconnects
                                                        are still
                                                        happening and
                                                        the number of
                                                        files in
                                                        split-brain is
                                                        growing.

                                                      Setup:
                                                        6 compute nodes,
                                                        each serving as
                                                        a glusterfs
                                                        server and
                                                        client, Ubuntu
                                                        14.04, two
                                                        bricks per node,
distribute-replicate

                                                      I
                                                        have two gluster
                                                        volumes set up
                                                        (one for scratch
                                                        data, one for
                                                        the slurm
                                                        scheduler). Only
                                                        the scratch data
                                                        volume shows
                                                        critical errors
                                                        "[...] has not
                                                        responded in the
                                                        last 42 seconds,
                                                        disconnecting.".
                                                        So I can rule
                                                        out network
                                                        problems, the
                                                        gigabit link
                                                        between the
                                                        nodes is not
                                                        saturated at
                                                        all. The disks
                                                        are almost idle
                                                        (<10%).

                                                      I
                                                        have glusterfs
                                                        3.4.2 on Ubuntu
                                                        12.04 on a
                                                        another compute
                                                        cluster, running
                                                        fine since it
                                                        was deployed.
                                                      I
                                                        had glusterfs
                                                        3.4.2 on Ubuntu
                                                        14.04 on this
                                                        cluster, running
                                                        fine for almost
                                                        a year.

                                                      After
                                                        upgrading to
                                                        3.8.5, the
                                                        problems (as
                                                        described)
                                                        started. I would
                                                        like to use some
                                                        of the new
                                                        features of the
                                                        newer versions
                                                        (like bitrot),
                                                        but the users
                                                        can't run their
                                                        compute jobs
                                                        right now
                                                        because the
                                                        result files are
                                                        garbled.

                                                          2016-11-29
                                                          18:53
                                                          GMT+01:00 Atin
                                                          Mukherjee <amukherj@xxxxxxxxxx>:

                                                          Would you be able to share what is not working for you in 3.8.x (mention the exact version). 3.4 is quite old and falling back to an unsupported version doesn't look a feasible option.

                                                          On
                                                          Tue, 29 Nov
                                                          2016 at 17:01,
                                                          Micha Ober
                                                          <micha2k@xxxxxxxxx>
                                                          wrote:

                                                          Hi,

                                                          I was using gluster 3.4 and
                                                          upgraded to
                                                          3.8, but that
                                                          version showed
                                                          to be unusable
                                                          for me. I now
                                                          need to
                                                          downgrade.

                                                          I'm running Ubuntu 14.04. As
                                                          upgrades of
                                                          the op version
are irreversible, I guess I have to delete all gluster volumes and
                                                          re-create them
                                                          with the
                                                          downgraded
                                                          version. 

                                                          0. Backup data
                                                          1. Unmount all gluster volumes
                                                          2. apt-get purge
                                                          glusterfs-server
glusterfs-client
                                                          3. Remove PPA for 3.8
                                                          4. Add PPA for older version
                                                          5. apt-get install
                                                          glusterfs-server
glusterfs-client
                                                          6. Create volumes

                                                          Is "purge" enough to delete all
                                                          configuration
                                                          files of the
                                                          currently
                                                          installed
                                                          version or do
                                                          I need to
                                                           manually
                                                          clear some
                                                          residues
                                                          before
                                                          installing an
                                                          older version?

                                                          Thanks.

                                                          _______________________________________________

                                                          Gluster-users
                                                          mailing list

                                                          Gluster-users@xxxxxxxxxxx

                                                          http://www.gluster.org/mailman/listinfo/gluster-users

                                                          --

                                                          -
                                                          Atin (atinm)

                                    _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

-- 

~ Atin (atinm)

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users