Re: replica performance and brick size best practice

[beer Ll <llcfhllml@xxxxxxxxx>]

Hi Strahil

thank you for your email

HW raid with HP Smart Array p840

8 disks SAS 8TB

RAID 6 -> 43,66TB

strip size: 128KB

full stripe size : 768KB

in this case no thin LVM

xfs with standard option of Debian isize=512 

no glusterfs tuning

no jumbo frames

I share the gluster volume with nfs (ganesha) , the application is on another server in LAN .

Best regards

On Sun, Nov 13, 2022 at 8:59 AM Strahil Nikolov <hunter86_bg@xxxxxxxxx> wrote:

Hi,

First you need to identify what kind of workload you will have.

Some optimizations for one workload can prevent better performance in another type.

If you plan to use the volume for small files , this document is a good start https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/small_file_performance_enhancements

Replica 2 volumes are prone to split brain and it's always recommended to have a replica 3 or an arbiter .

As a best practice - always test the volume with the application that will use it as synthetic benchmarks are just synthetics.

I always start performance review from the bottom (infrastructure) and end on application level.

You can start with https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance as the storage is one of the most important parts.

What kind of HW raid , how many disks, stripe size and stripe width that you used in LVM?

Do you use thin LVM ?

How did you create your XFS (isize is critical) ?

Have you used gluster's tuned profile ?

Jumbo frames ?

Then you will need to optimize the volume for small files (see the link above).

Does you app allow you to use libgfapi ? Based on my observations in the oVirt list, libgfapi it used to provide some performance benefits compared to fuse .

Also, if you work with very small files, it would make sense to combine them in some container (like in VM disks).

Keep in mind that GlusterFS performance scales with the size of the cluster and the number of clients. For ultra high performance for a few clients -> there are other options.

Best Regards,

Strahil Nikolov 

On Wed, Nov 9, 2022 at 12:05, beer Ll

<llcfhllml@xxxxxxxxx> wrote:

Hi

I have 2 gluster server (backup1 , backup2)  connected with 10Gbit link

glusterfs version 10.3-1  server and client

Each server is  with 44T disk raid 6 with 1 partition used for the brick

 /dev/VG_BT_BRICK1/LV_BT_BRICK1   [      43.66 TiB]

/dev/mapper/VG_BT_BRICK1-LV_BT_BRICK1 on /mnt/glusterfs type xfs (rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=1536,noquota)

I create a replica volume named backup

Volume Name: backup
Type: Replicate
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: backup-1:/mnt/glusterfs/brick1
Brick2: backup-2:/mnt/glusterfs/brick1
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
cluster.enable-shared-storage: enable

The volume backup is mounted with gluster client on /mnt/share

backup-1:/backup on /mnt/share type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

Test with smallfile utility

XFS

with filesystem xfs on  /mnt/glusterfs

total threads = 8
total files = 800
100.00% of requested files processed, warning threshold is  70.00%
elapsed time =     0.009
files/sec = 120927.642211

GLUSTERFS CLIENT

with  glusterfs on /mnt/share

total threads = 8
total files = 800
100.00% of requested files processed, warning threshold is  70.00%
elapsed time =     3.014
files/sec = 284.975861

How is it possible to increase the performance in glusterfs volume ?

How is best practice of brick size and replica management ?

Is better 1 big brick per server or more small bricks distributed ?

Many Thanks

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users