Re: GFS performance under heavy traffic

[Strahil <hunter86_bg@xxxxxxxxx>]

As your issue is Network, consider changing the MTU if the infrastructure is allowing it.
The tuned profiles are very important, as they control ratios for dumping data in memory to disk (this case gluster over network). You want to avoid keeping a lot of data in client's memory(in this  case  the gluster server), just to unleash it over network.

These 2 can be implemented online and I do not expect any issues.

Filesystem of bricks is important because the faster they soak data, the faster gluster can take more.

Of  course,  you need to reproduce it in  test.

Also  consider checking if there  is  any kind  of backup  running on the bricks. I have seen too many 'miracles' :D

Best Regards,
Strahil Nikolov

On Jan 8, 2020 01:03, David Cunningham <dcunningham@xxxxxxxxxxxxx> wrote:

Hi Strahil,

Thanks for that. The queue/scheduler file for the relevant disk reports "noop [deadline] cfq", so deadline is being used. It is using ext4, and I've verified that the MTU is 1500.

We could change the filesystem from ext4 to xfs, but in this case we're not looking to tinker around the edges and get a small performance improvement - we need a very large improvement on the 114MBps of network traffic to make it usable.

I think what we really need to do first is to reproduce the problem in testing, and then come back to possible solutions.

On Tue, 7 Jan 2020 at 22:15, Strahil Nikolov <hunter86_bg@yahoo.com> wrote:

To find the scheduler , find all pvs of the LV is providing your storage

[root@ovirt1 ~]# df -Th /gluster_bricks/data_fast
Filesystem                                       Type  Size  Used Avail Use% Mounted on
/dev/mapper/gluster_vg_nvme-gluster_lv_data_fast xfs   100G   39G   62G  39% /gluster_bricks/data_fast

[root@ovirt1 ~]# pvs | grep gluster_vg_nvme
  /dev/mapper/vdo_nvme gluster_vg_nvme lvm2 a--  <1024.00g    0

[root@ovirt1 ~]# cat /etc/vdoconf.yml
####################################################################
# THIS FILE IS MACHINE GENERATED. DO NOT EDIT THIS FILE BY HAND.
####################################################################
config: !Configuration
  vdos:

   vdo_nvme: !VDOService

      device: /dev/disk/by-id/nvme-ADATA_SX8200PNP_2J1120011596

[root@ovirt1 ~]# ll /dev/disk/by-id/nvme-ADATA_SX8200PNP_2J1120011596
lrwxrwxrwx. 1 root root 13 Dec 17 20:21 /dev/disk/by-id/nvme-ADATA_SX8200PNP_2J1120011596 -> ../../nvme0n1
[root@ovirt1 ~]# cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber

Note: If device is under multipath , you need to check all paths (you can get them from 'multipath -ll' command).

The only scheduler you should avoid is "cfq" which was default for RHEL 6 & SLES 11.

XFS has better performance that ext-based systems.

Another tuning is to use Red hat's tuned profiles for gluster. You can extract them from (or newer if you find) ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.4.2.0-1.el7rhgs.src.rpm

About MTU - it's reducing the ammount of packages that the kernel has to process - but requires infrastructure to support that too. You can test by setting MTU on both sides to 9000 and then run 'tracepath remote-ip'. Also run a ping with large size without do not fragment flag ->  'ping -M do -s 8900 <

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users