Hello all,
I'm setting up GlusterFS on 2 hw w/ same configuration, 8 hdds. This deployment will grow later on.
I'm undecided between these different configurations and am seeing comments or advice from more experienced users of GlusterFS.
Here is the summary of 3 options:
1. 1 brick per host, Gluster "distributed" volumes, internal redundancy at brick level
# 1 brick per host, simplified cluster management, higher blast-radius
having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid10 of 8 hdd.
Pros:
* I know ZFS raid10 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster in "distributed" mode, no replication (is this a pro?)
* large blast radius, if a zfs volume goes bad or node goes bad, I lose data.
* not using "distributed replicated" (is this a con?)
* I can't use hosts without internal redundancy later on?
# 1 brick per hard disk, fine grained device management on Gluster, smaller blast-radius.
Having 1 brick per drive (/data/bricks/hddN for 1 to X drives on box), each brick would still use ZFS.
Pros:
* 1 drive blast radius, the ideal.
* GlusterFS w/ distributed replicated
* no complicated host-fault management or runbook, I can use hosts with low availability
Cons:
* distributed replicated performance vs zfs raid10
* managing on gluster at the disk level can be more time consuming
* managing disk spares and replacements w/ gluster
having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid0 of 8 hdd.
Pros:
* I know ZFS raid0 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster for redundancy (is this a pro?)
I'm setting up GlusterFS on 2 hw w/ same configuration, 8 hdds. This deployment will grow later on.
I'm undecided between these different configurations and am seeing comments or advice from more experienced users of GlusterFS.
Here is the summary of 3 options:
1. 1 brick per host, Gluster "distributed" volumes, internal redundancy at brick level
2. 1 brick per drive, Gluster "distributed replicated" volumes, no internal redundancy
3. 1 brick per host, Gluster "distributed replicated" volumes, no internal redundancy
I don't know how the performance of these different configurations would compare. The workload is HPC and ML-training, data-heavy, metadata-light.
having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid10 of 8 hdd.
Pros:
* I know ZFS raid10 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster in "distributed" mode, no replication (is this a pro?)
* don't need to worry about GlusterFS performance with "distributed replicated"
* future hosts can have raidz2 + hot spares or whatnot
Cons:* large blast radius, if a zfs volume goes bad or node goes bad, I lose data.
* not using "distributed replicated" (is this a con?)
* I can't use hosts without internal redundancy later on?
# 1 brick per hard disk, fine grained device management on Gluster, smaller blast-radius.
Having 1 brick per drive (/data/bricks/hddN for 1 to X drives on box), each brick would still use ZFS.
Pros:
* 1 drive blast radius, the ideal.
* GlusterFS w/ distributed replicated
* no complicated host-fault management or runbook, I can use hosts with low availability
Cons:
* distributed replicated performance vs zfs raid10
* managing on gluster at the disk level can be more time consuming
* managing disk spares and replacements w/ gluster
# 1 brick per host, coarse grained brick management on Gluster, Gluster for replication.
having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid0 of 8 hdd.
Pros:
* I know ZFS raid0 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster for redundancy (is this a pro?)
* system can use lower redundancy hosts
Cons:
* large blast radius, replicating a whole host takes longer.
Cons:
* large blast radius, replicating a whole host takes longer.
Any comments or advice is welcome
Thanks in advance,
-- Miguel Filipe
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
