Hi Dan, I'm not kidding, these were real-world observations, hence my motivation to create this balancer :) First I tried "fixing" the mgr balancer, but after understanding the exact algorithm there I thought of a completely different approach. For us the main reason things got out of balance was this (from the README): > To make things worse, if there's a huge server in the cluster which is so big, CRUSH can't place data often enough on it to fill it to the same level as any other server, the balancer will fail moving PGs across servers that actually would have space. > This happens since it sees only this server's OSDs as "underfull", but each PG has one shard on that server already, so no data can be moved on it. But all the aspects in that section play together, and I don't think it's easily improvable in mgr-balancer while keeping the same base algorithm. Cheers -- Jonas On 20/10/2021 19.55, Dan van der Ster wrote: > Hi Jonas, > > From your readme: > > "the best possible solution is some OSDs having an offset of 1 PG to the ideal count. As a PG-distribution-optimization is done per pool, without checking other pool's distribution at all, some devices will be the +1 more often than others. At worst one OSD is the +1 for each pool in the cluster." > > That's an interesting observation/flaw which hadn't occurred to me before. I think we don't ever see it in practice in our clusters because we do not have multiple large pools on the same osds. > > How large are the variances in your real clusters? I hope the example in your readme isn't from real life?? > > Cheers, Dan _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
