Re: Advice for setup: SW RAID 6 vs JBOD

[Jim Kinney <jim.kinney@xxxxxxxxx>]

I have about 200TB in a gluster replicate only 3-node setup. We stopped using hardware RAID6 after the third drive failed on one array at the same time we replaced the other two and before recovery could complete. 200TB is a mess to resync.

So now each hard drive is a single entity. We add 1 drive to each node as it's own PV in gluster (with LUKS encryption). Each brick is mounted into the final tree on the client end.

This way our recover is usually just a single drive to sync. With replica 3, we keep quorum if one brick fails. No RAID cards. Just big, multipath SAS JBOD arrays. The server head on each array is pretty beefy (24 cores, 128GB RAM, 40G IB, 40G Ethernet).

On Thu, 2019-06-06 at 20:46 +0200, Michael Metz-Martini wrote:

Hi

Am 06.06.19 um 18:48 schrieb Eduardo Mayoral:

Your comment actually helps me more than you think, one of the main

doubts I have is whether I go for JOBD with replica 3 or SW RAID 6 with

replica2 + arbitrer. Before reading your email I was leaning more

towards JOBD, as reconstruction of a moderately big RAID 6 with mdadm

can be painful too. Now I see a reconstruct is going to be painful

either way...

For the record, the workload I am going to migrate is currently

18,314,445 MB and 34,752,784 inodes (which is not exactly the same as

files, but let's use that for a rough estimate), for an average file

size of about 539 KB per file.

Thanks a lot for your time and insights!

Currently we're hosting ~200 TB split into about 3.500.000.000 files on

a Distributed-Replicate-2-gluster volume with each brick running on a

hw-raid6 of 8 x 8 TB disks. As we never had a failed drive 'till now I

can't tell you something about recovery times but rebalance is damn slow

with such high number of small files (so should recovery on

jbod-bricks). I think raid-recovery from local disks will be much faster.

As our files are nearly 100% readonly and split-brain-issues could be

resolevd more or less "easily" we decided against replica 3 in favor of

hardware raid6 redundancy.

-- 
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users