Have you tested increasing:
- performance.cache-size
- performance.write-behind-window-size
Also, you should note that the artificial benchmarks usually do not cover your needs. The best way is to test with real-world workload, as tuning for specific workload is reducing the performance for another workload.
For example, optimizing for sequential I/O is requiring options that are opposite to random I/O patterns.
For small file optimizations, you can check https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/small_file_performance_enhancements
I/O workloads can be optimized via 2 tuned profiles from http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.5.0.0-7.el7rhgs.src.rpm
Also you can tune gluster towards your workload by using the gluster settings in /var/lib/glusterd/groups (I wrote it by memory, so check for typos).
In order to get it further debugged, you have to 'profile' your volume as per https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-monitoring_red_hat_storage_workload and share the output.
Best Regards,
Strahil Nikolov
On Sat, May 15, 2021 at 7:45, Schlick Rupert<Rupert.Schlick@xxxxxxxxx> wrote:Dear Felix,
as requested, volume info, xfs_info, fstab.
Volume Name: gfs_scratch
Type: Replicate
Volume ID: d99b6154-bf34-49d6-a06b-0e29bfc2a0fb
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: server3:/data/glusterfs_scratch/gfs_scratch_brick1
Brick2: server2:/data/glusterfs_scratch/gfs_scratch_brick1
Brick3: server1:/data/glusterfs_scratch/gfs_scratch_brick1
Options Reconfigured:
performance.parallel-readdir: on
cluster.self-heal-daemon: enable
features.uss: disable
cluster.server-quorum-type: none
performance.cache-size: 256MB
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 200000
performance.write-behind-window-size: 128MB
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
snap-activate-on-create: enable
auto-delete: enable
$ xfs_info /data/glusterfs_scratch
meta-data="" isize=512 agcount=16, agsize=3276768 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=52428288, imaxpct=25
= sunit=32 swidth=128 blks
naming =version 2 bsize=8192 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=25599, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# /etc/fstab: static file system information.
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/disk/by-id/dm-uuid-LVM-IHnHtvssE4vRhHeAkSbMX7DDF8uwnlsZsaQvW2bySOYnc17QGlT7FiESTD9GloaL / ext4 defaults 0 0
/dev/disk/by-uuid/7aa78d34-0c19-47dc-85e2-70b54cfb9868 /boot ext4 defaults 0 0
/dev/disk/by-uuid/37A6-0D34 /boot/efi vfat defaults 0 0
/swap.img none swap sw 0 0
UUID=f070d8a9-ade5-451f-9bf6-53de6c1a3789 /data/glusterfs_home xfs inode64,noatime,nodiratime 0 0
UUID=187654ce-99f7-4aea-b2f6-701cea801b01 /data/glusterfs_sw xfs inode64,noatime,nodiratime 0 0
UUID=1176552b-7233-4354-be32-a6dc0e899d64 /data/glusterfs_scratch xfs inode64,noatime,nodiratime 0 0
server1:/gfs_home /home/gfs glusterfs defaults,_netdev 0 0
server1:/gfs_sw /sw glusterfs defaults,_netdev 0 0
server1:/gfs_scratch /scratch glusterfs defaults,_netdev 0 0
/dev/mapper/GVG-gitlab--builds /mnt/gitlab-builds ext4 defaults,noatime,nodiratime 0 0
Cheers
Rupert
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
