Hi there,
We've done some pretty extensive testing on our new cluster of 11 x (24x18TB HDD, 2x2.9TB NVMe) nodes for erasure code. We were particularly interested in the some of the alternate plugins like CLAY and LRC, however I came across some really curious behaviour that I couldn't find an answer to.
In my testing, the added helper chunks for LRC ("l") and CLAY ("d") are still run through CRUSH like "normal" k or m chunks. So, for example, pgs will always be undersized in an 11 node cluster (host fd) with k=8,m=2, l=5 (12 chunks > 11 nodes). However, in my testing, the additional "helper chunks" (either in CLAY or LRC) do not add any significant overhead storage penalty (beyond a rounding error, at least).
In this cluster, I also saw no clear benefit either for write or recovery for either LRC or CLAY, but I only used ~20TiB (before overhead) test data using rados bench, so perhaps some of these specialty plugins really start shining at scale? (And they're not the same use case, I understand, ie from my reading LRC really benefits recovery in multi-rack installations where one rack (evenly split) doesn't need to communicate with the other, assuming the matching CRUSH hierarchy is in place).
Anyone have any good resources on this beyond the documentation, or at a minimum can explain or confirm the slightly spooky nature of the "helper chunks" mentioned above?
With thanks,
Sean Matheny
HPC Cloud Platform DevOps Lead
New Zealand eScience Infrastructure (NeSI)
e: sean.matheny@xxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
