Best Regards,
Strahil Nikolov
On Sunday, April 23, 2023, 2:35 AM, Peter P <peter.a.pfeiffer@xxxxxxxxx> wrote:
________Good afternoon,
I am looking for additional information about glusterfs, arbiters and thin-arbiters. The current stable release is gluster 11, so I am considering that version for deployment. My planned setup is: 4 storage servers in distributed + replicated mode.
Server1, server2 : replica 2, arbiter 1
Server3, server4 : replica 2, arbiter 1
Since having replica 2 is not recommended due to split-brains I will have an arbiter.
Generic questions:
- Is arbiter or thin-arbiter recommended in a production, large volume storage environment?
- Both were introduced a long time ago. Most users prefer full arbiter, so healing is far more optimal (only changed files will be healed).
- Is thin arbiter code stable and deployment ready?
- I know that it’s in use but full arbiter has been introduced earlier and has a wider adoption.
- Arbiter is file based and stores metadata for each file. In this scenario I would at least double the default inode count on the arbiter server. Thin-arbiter on the other hand is brick based but I have not found enough information if its inode usage is as inode hungry as the arbiter configuration. I am thinking, it should not be as it is brick based. So do I need to increase the inode count when using thin-arbiters? If yes, what is recommended?
- Full arbiter is sensitive to network latency and disk speed (a lot of small I/O for those inodes). Increase macpct (XFS) on arbiter bricks and prefer using a SSD/NVME. As full arbiter doesn’t store any data , you can set the maxpct to around 75%- Thin arbiter doesn’t have a brick and when you create it, you just specify the replica id file ( see https://docs.gluster.org/en/main/Administrator-Guide/Thin-Arbiter-Volumes/ )
- I've read old bug reports reporting that thin arbiter was not ready to serve multiple trusted pools. Is this still the case? I may configure multiple trusted pools in the future.
- I saw Kadalu uses their own thin arbiter and I never saw issues. I doubt I was the only one using it, so it should be fine.
- I have many linux boxes running different linux distributions and releases. Ideally the assortment of boxes would mount the same gluster pool/volume. I looked for information about older versions of gluster clients running on a range of older distributions mounting the most recent gluster 11 pool/volume? Does that work? Can gluster client (version 10, 9, 8, 7, etc.) mount gluster 11 volume and run without significant issues? I understand that older versions of client will not have the most recent features. Most recent features aside, is such configuration supported/stable?
- For that purpose gluster has 2 settings:cluster.max-op-version -> the max compatibility version you can set your cluster based of the oldest client’s versioncluster.op-version -> the cluster’s compatibility versionAs long you keep the cluster.op-version compatible with your client - it should work.
Thin-arbiter approach: If I go with the thin-arbiter configuration I will use a 5th server as this server can be outside of the trusted pool and can be shared among multiple trusted pools
Server1, server2: replica 2, thin-arbiter server5
Server3, server4: replica 2, thin-arbiter server5
Old arbiter approach: If I go with the older arbiter configuration, I am considering using 2 of the storage servers to act as both replica and an arbiter. Is that configuration possible/supported and reasonable?
Server1, server2: replica 2, arbiter server3
Server3, server4: replica 2, arbiter server1
- Yes, as long as you have a dedicated brick (in this example server3 should have a data brick and arbiter brick)
In this configuration, I am considering using server3 to be arbiter for server{1,2} replica 2, and using server1 to be arbiter for server{3,4} replica 2.
Questions:
- Is this a reasonable/recommended configuration?
- It’s used quite often- Always keep bricks as separate mount points. For example:
- Should the arbiter metadata folder be inside or outside of the volume?
- In detail. Say server{1,2} replica has 1 brick each /gluster/brick1 with /gfs1vol1 as the volume
- Should the arbiter metadata folder location be: /gluster/arbiter/gfs1vol1 (outside of the volume path) or /gfs1vol1/arbiter1/ (inside the volume path)
/dev/vg/lv mounted on /bricks/databricks with directory vol1/brick1/dev/vg/lv2 mounted on /bricks/arbiterbricks with directory vol1/arbiterbrick1The idea is that if the device is not mounted, the brick directory will be missing and the mess will be far less.
Thank you for your thoughts,
Peter
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
