gluster performance issue

"paf1@xxxxxxxx" <paf1@xxxxxxxx> · Sat, 28 Jan 2017 10:16:03 +0100



    Hello, 

      can anybody help with gluster performace ?

      
      I'm running gluster replica 3 arbiter 1 ( 2+1) with following
      version  

    
                  OS Version:RHEL - 7 - 3.1611.el7.centos
                
              
                  OS Description:CentOS Linux 7 (Core)
                
              
                  Kernel Version:3.10.0 - 514.6.1.el7.x86_64
                
              
                  KVM Version:2.6.0 - 28.el7_3.3.1
                
              
                  LIBVIRT Version:libvirt-2.0.0-10.el7_3.4
                
              
                  VDSM Version:vdsm-4.18.21-1.el7.centos
                
              
                  SPICE Version:0.12.4 - 19.el7
                
              
                  GlusterFS Version:glusterfs-3.7.19-1.el7
                
              
                  CEPH Version:librbd1-0.94.5-1.el7

                      
                      After huge gluster load ( runnig DB + 2x import
                      over 10Gb Lan ) system generate messages to
                      "messages"

                      and dramaticaly goes down, even node rebooted .

                      
                      Jan 28 09:45:19 1kvm2 kernel: INFO: task
                      worker:11384 blocked for more than 120 seconds.

                      Jan 28 09:45:19 1kvm2 kernel: "echo 0 >
                      /proc/sys/kernel/hung_task_timeout_secs" disables
                      this message.

                      Jan 28 09:45:19 1kvm2 kernel: worker          D
                      ffff880167566dd0     0 11384      1 0x00000080

                      Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76fa60
                      0000000000000086 ffff882fba75de20 ffff8807ab76ffd8

                      Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76ffd8
                      ffff8807ab76ffd8 ffff882fba75de20 ffff882fbf396c40

                      Jan 28 09:45:19 1kvm2 kernel: 0000000000000000
                      7fffffffffffffff ffff882fa8ee2500 ffff8807ab76fd60

                      Jan 28 09:45:19 1kvm2 kernel: Call Trace:

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff8168b899>] schedule+0x29/0x70

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff816892e9>]
                      schedule_timeout+0x239/0x2d0

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffffa0889ea2>] ?
                      fuse_direct_io+0x492/0x820 [fuse]

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff8168ae3e>]
                      io_schedule_timeout+0xae/0x130

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff8168aed8>] io_schedule+0x18/0x20

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff8124d945>]
                      wait_on_sync_kiocb+0x35/0x80

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffffa088b091>]
                      fuse_direct_IO+0x231/0x380 [fuse]

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffffa088d1f8>] ?
                      fuse_change_attributes_common+0x88/0x120 [fuse]

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffffa088d32f>] ?
                      fuse_change_attributes+0x9f/0x100 [fuse]

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff8118283c>]
                      generic_file_aio_read+0x70c/0x790

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffffa0887d78>]
                      fuse_file_aio_read+0x78/0xb0 [fuse]

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff811fdad9>]
                      do_sync_readv_writev+0x79/0xd0

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff811ff17e>]
                      do_readv_writev+0xce/0x260

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff810f7bae>] ? do_futex+0xfe/0x5b0

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff811ff342>] vfs_readv+0x32/0x60

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff811ff6b2>] SyS_preadv+0xc2/0xf0

                      Jan 28 09:45:19 1kvm2 kernel:
                      [<ffffffff816967c9>]
                      system_call_fastpath+0x16/0x1b

                      Jan 28 09:50:01 1kvm2 systemd: Started Session 58
                      of user root.

                      Jan 28 09:50:01 1kvm2 systemd: Starting Session 58
                      of user root.

                      
                      I tried to set kernel params via web workaround
                      recommendations, but without any success

                      ( current /etc/sysctl.conf settings )

                      vm.swappiness=1 

                      vm.vfs_cache_pressure=100

                      vm.min_free_kbytes = 512000

                      vm.zone_reclaim_mode = 1

                      
                      Any idea ??

                      regs.

                      Paf1

                    
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users