Re: glusterfs under high load failing?

Roman <romeo.r@xxxxxxxxx> · Mon, 13 Oct 2014 19:30:02 +0300

Sure.I'll let it to run for this night .

2014-10-13 19:19 GMT+03:00 Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>:

    hi Roman,

         Do you think we can run this test again? this time, could you
    enable 'gluster volume profile <volname> start', do the same
    test. Provide output of 'gluster volume profile <volname>
    info' and logs after the test?

    Pranith

    On 10/13/2014 09:45 PM, Roman wrote:

      Sure !

          root@stor1:~# gluster volume info

          Volume Name: HA-2TB-TT-Proxmox-cluster
          Type: Replicate
          Volume ID: 66e38bde-c5fa-4ce2-be6e-6b2adeaa16c2
          Status: Started
          Number of Bricks: 1 x 2 = 2
          Transport-type: tcp
          Bricks:
          Brick1: stor1:/exports/HA-2TB-TT-Proxmox-cluster/2TB
          Brick2: stor2:/exports/HA-2TB-TT-Proxmox-cluster/2TB
          Options Reconfigured:
          nfs.disable: 0
          network.ping-timeout: 10

          Volume Name: HA-WIN-TT-1T
          Type: Replicate
          Volume ID: 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
          Status: Started
          Number of Bricks: 1 x 2 = 2
          Transport-type: tcp
          Bricks:
          Brick1: stor1:/exports/NFS-WIN/1T
          Brick2: stor2:/exports/NFS-WIN/1T
          Options Reconfigured:
          nfs.disable: 1
          network.ping-timeout: 10

        2014-10-13 19:09 GMT+03:00 Pranith
          Kumar Karampuri <pkarampu@xxxxxxxxxx>:

             Could you give your
              'gluster volume info' output?

              Pranith

                  On 10/13/2014 09:36 PM, Roman wrote:

                    Hi,

                      I've got this kind of setup (servers run
                        replica)

                      @ 10G backend
                      gluster storage1
                      gluster storage2
                      gluster client1

                      @1g backend
                      other gluster clients

                      Servers got HW RAID5 with SAS disks.

                      So today I've desided to create a 900GB file
                        for iscsi target that will be located @
                        glusterfs separate volume, using dd (just a
                        dummy file filled with zeros, bs=1G count 900)
                      For the first of all the process took pretty
                        lots of time, the writing speed was 130 MB/sec
                        (client port was 2 gbps, servers ports were
                        running @ 1gbps).
                      Then it reported something like "endpoint is
                        not connected" and all of my VMs on the other
                        volume started to give me IO errors.
                      Servers load was around 4,6 (total 12 cores)

                      Maybe it was due to timeout of 2 secs, so
                        I've made it a big higher, 10 sec.

                      Also during the dd image creation time, VMs
                        very often reported me that their disks are slow
                        like

                        WARNINGs: Read IO Wait time is -0.02 (outside
                          range [0:1]).
                        Is 130MB /sec is the maximum bandwidth for
                          all of the volumes in total? That why would we
                          need 10g backends?
                        HW Raid local speed is 300 MB/sec, so it
                          should not be an issue. any ideas or mby any
                          advices?

                        Maybe some1 got optimized sysctl.conf for 10G
                          backend?
                        mine is pretty simple, which can be found
                          from googling.

                        just to mention: those VM-s were connected
                          using separate 1gbps intraface, which means,
                          they should not be affected by the client with
                          10g backend.

                        logs are pretty useless, they just say  this
                          during the outage

                        [2014-10-13 12:09:18.392910] W
                          [client-handshake.c:276:client_ping_cbk]
                          0-HA-2TB-TT-Proxmox-cluster-client-0: timer
                          must have expired
                        [2014-10-13 12:10:08.389708] C
                          [client-handshake.c:127:rpc_client_ping_timer_expired]
                          0-HA-2TB-TT-Proxmox-cluster-client-0: server 10.250.0.1:49159 has not
                          responded in the last 2 seconds,
                          disconnecting.
                        [2014-10-13 12:10:08.390312] W
                          [client-handshake.c:276:client_ping_cbk]
                          0-HA-2TB-TT-Proxmox-cluster-client-0: timer
                          must have expired

                      so I decided to set the timout a bit higher.

                        So it seems to me, that under high load
                          GlusterFS is not useable? 130 MB/s is not that
                          much to get some kind of timeouts or makeing
                          the systme so slow, that VM-s feeling
                          themselves bad.

                        Of course, after the disconnection, healing
                          process was started, but as VM-s lost
                          connection to both of servers, it was pretty
                          useless, they could not run anymore. and BTW,
                          when u load the server with such huge job (dd
                          of 900GB), healing process goes soooooo slow
                          :)

                        -- 

                        Best regards,

                        Roman. 

                _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

        -- 

        Best regards,

        Roman.

-- 
Best regards,
Roman.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users