Re: Crushmap rule for multi-datacenter erasure coding

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Frank,

Thanks for this detailed answer. About your point of 4+2 or similar schemes defeating the purpose of a 3-datacenter configuration, you're right in principle. In our case, the goal is to avoid any impact for replicated pools (in particular RBD for the cloud) but it may be acceptable for some pools to be readonly during a short period. But I'll explore your alternative k+m scénarios as some may be interesting..

I'm also interested by experience feedback with LRC EC, even if I don't think it changes the problem for resilience to a DC failure.

Best regards,

Michel
Sent from my mobile
Le 3 avril 2023 21:57:41 Frank Schilder <frans@xxxxxx> a écrit :

Hi Michel,

failure domain = datacenter doesn't work, because crush wants to put 1 shard per failure domain and you have 3 data centers and not 6. The modified crush rule you wrote should work. I believe equally well with x=0 or 2 -- but try it out before doing anything to your cluster.

The easiest way for non-destructive testing is to download the osdmap from your cluster and from that map extract the crush map. You can then *without* modifying your cluster update the crush map in the (off-line) copy of the OSD map and let it compute mappings (commands for all this are in the ceph docs, look for osdmaptool). These mappings you can check for if they are as you want. There was an earlier case where someone posted a script to confirm mappings automatically. I used some awk magic, its not that difficult.

As a note of warning, if you want to be failure resistant, don't use 4+2. Its not worth the effort of having 3 data centers. In case you loose one DC, you have only 4 shards left, in which case the pool becomes read-only. Don't even consider to set min_size=4, it again completely defeats the purpose of having 3 DCs in the first place.

The smallest profile you can use that will ensure RW access in case of a DC failure is 5+4 (55% usable capacity). If 1 DC fails, you have 6 shards, which is equivalent to 5+1. Here, you have RW access in case of 1 DC down. However, k=5 is a prime number with negative performance impact, ideal are powers of 2 for k. The alternative is k=4, m=5 (44% usable capacity) with good performance but higher redundancy overhead.

You can construct valid schemes by looking at all N multiples of 3 and trying k<=(2N/3-1):

N=6 -> k=2 m=4
N=9 -> k=4, m=5 ; k=5, m=4
N=12 -> k=6, m=6; k=7, m=5
N=15 -> k=8, m=7 ; k=9, m=6

As you can see, the larger N the smaller the overhead. The downside is larger stripes, meaning that larger N only make sense if you have a large files/objects. An often overlooked advantage of profiles with large m is that you can defeat tail latencies for read operations by setting fast_read=true for the pool. This is really great when you have silently failing disks. Unfortunately, there is no fast_write counterpart (which would not be useful in your use case any ways).

There are only very few useful profiles with k a power of 2 (4+5, 8+7). Some people use 7+5 with success and 5+4 does look somewhat OK as well. If you use latest ceph with bluestore min alloc size = 4K, stripe size is less of an issue and 8+7 is a really good candidate that I would give a shot in a benchmark.

You should benchmark a number of different profiles on your system to get an idea of how important the profile is for performance and how much replication overhead you can afford. Remember to benchmark also in degraded condition. While as an admin you might be happy that stuff is up, users will still complain if things are suddenly unbearably slow. Make long-running tests in degraded state to catch all the pitfalls of MONs not trimming logs etc. to have a reliable configuration that doesn't let you down the first time it rains.

Good luck and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Michel Jouvin <michel.jouvin@xxxxxxxxxxxxxxx>
Sent: Monday, April 3, 2023 6:40 PM
To: ceph-users@xxxxxxx
Subject:  Crushmap rule for multi-datacenter erasure coding

Hi,

We have a 3-site Ceph cluster and would like to create a 4+2 EC pool
with 2 chunks per datacenter, to maximise the resilience in case of 1
datacenter being down. I have not found a way to create an EC profile
with this 2-level allocation strategy. I created an EC profile with a
failure domain = datacenter but it doesn't work as, I guess, it would
like to ensure it has always 5 OSDs up (to ensure that the pools remains
R/W) where with a failure domain = datacenter, the guarantee is only 4.
My idea was to create a 2-step allocation and a failure domain=host to
achieve our desired configuration, with something like the following in
the crushmap rule:

step choose indep 3 datacenter
step chooseleaf indep x host
step emit

Is it the right approach? If yes, what should be 'x'? Would 0 work?

From what I have seen, there is no way to create such a rule with the
'ceph osd crush' commands: I have to download the current CRUSHMAP, edit
it and upload the modified version. Am I right?

Thanks in advance for your help or suggestions. Best regards,

Michel

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux