Erasure code ruleset for small cluster

[Caspar Smit <casparsmit@xxxxxxxxxxx>]

Hi all,

I'd like to setup a small cluster (5 nodes) using erasure coding. I would like to use k=5 and m=3.

Normally you would need a minimum of 8 nodes (preferably 9 or more) for this.

Then i found this blog: https://ceph.com/planet/erasure-code-on-small-clusters/

This sounded ideal to me so i started building a test setup using the 5+3 profile

Changed the erasure ruleset to:

rule erasure_ruleset {
  ruleset X
  type erasure
  min_size 8
  max_size 8
  step take default
  step choose indep 4 type host
  step choose indep 2 type osd
  step emit
}

Created a pool and now every PG has 8 shards in 4 hosts with 2 shards each, perfect.

But then i tested a node failure, no problem again, all PG's stay active (most undersized+degraded, but still active). Then after 10 minutes the OSD's on the failed node were all marked as out, as expected.

I waited for the data to be recovered to the other (fifth) node but that doesn't happen, there is no recovery whatsoever.

Only when i completely remove the down+out OSD's from the cluster the data is recovered.

My guess is that the "step choose indep 4 type host" chooses 4 hosts beforehand to store data on.

Would it be possible to do something like this:

Create a 5+3 EC profile, every hosts has a maximum of 2 shards (so 4 hosts are needed), in case of node failure -> recover data from failed node to fifth node.

Thank you in advance,

Caspar

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com