Fwd: New GlusterFS deployment, doubts on 1 brick per host vs 1 brick per drive.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello all,

I'm setting up GlusterFS on 2 hw w/ same configuration, 8  hdds. This deployment will grow later on.

I'm undecided between these different configurations and am seeing comments or advice from more experienced users of GlusterFS.

Here is the summary of 3 options:
1. 1 brick per host, Gluster "distributed" volumes, internal redundancy at brick level
2. 1 brick per drive, Gluster "distributed replicated" volumes, no internal redundancy
3. 1 brick per host, Gluster "distributed replicated" volumes, no internal redundancy

I don't know how the performance of these different configurations would compare. The workload is HPC and ML-training, data-heavy, metadata-light.


# 1 brick per host, simplified cluster management, higher blast-radius

having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid10 of 8 hdd.
Pros:
* I know ZFS raid10 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster in "distributed" mode, no replication (is this a pro?)
* don't need to worry about GlusterFS performance with "distributed replicated"
* future hosts can have raidz2 + hot spares or whatnot
Cons:
* large blast radius, if a zfs volume goes bad or node goes bad, I lose data.
* not using "distributed replicated" (is this a con?)
* I can't use hosts without internal redundancy later on?


# 1 brick per hard disk, fine grained device management on Gluster, smaller blast-radius.

Having 1 brick per drive (/data/bricks/hddN for 1 to X drives on box), each brick would still use ZFS.
Pros:
* 1 drive blast radius, the ideal.
* GlusterFS w/ distributed replicated
* no complicated host-fault management or runbook, I can use hosts with low availability
Cons:
* distributed replicated performance vs zfs raid10
* managing on gluster at the disk level can be more time consuming
* managing disk spares and replacements w/ gluster


# 1 brick per host, coarse grained brick management on Gluster, Gluster for replication.

having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid0 of 8 hdd.

Pros:
* I know ZFS raid0 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster for redundancy (is this a pro?)
* system can use lower redundancy hosts
Cons:
* large blast radius, replicating a whole host takes longer.

Any comments or advice is welcome


Thanks in advance,

-- Miguel Filipe
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux