On 6/29/22 14:58, Frank Schilder wrote:
Dear Stefan,
this looks like a terrible "improvement" as it implies a large number of redundant object movements together with an unnecessarily and hugely prolonged state of rebalancing.
It depends who you ask I guess :-). For many (most?) clusters this might
be preferable over old behavior. But it totally depends on workload,
hardware capabilities, Ceph version, etc.
So far I always disabled rebalancing/recovery, added new OSDs, increased
PG[P]_num, waited for peering and let ceph loose. Everything was
distributed and went to the right place in one go and was finished after
2-3 weeks (ca. 1000 OSDs now).
Is there really no way to do this in an atomic operation any more? Would target_max_misplaced_ratio=100% do the trick?
If you decrease pgp_num with for example 1 PG, have it remap /
rebalance, and after that increase pg_num ... and pgp_num to the same
number, it will be set instantly. At least that is IIRC from my
experience (I was really surprised by this behavior as well). It's there
to reduce the impact of mass data movement in the cluster.
I would like to have a "knob" to overrule this behavior, or a
--i-really-mean-it option to restore old behavior, and have a choice as
operator.
Maybe target_max_misplaced_ratio=100 would do the trick. I have not
tried that yet.
We use CERNs "upmap-remapped.py" to quickly have Ceph in HEALTH_OK, and
have the balancer remove upmaps to move data slowly. Dan and I have
agreed upon writing a "story" how this should work in practice (not yet
done). And instead of a (smart) trick, have it as default in the
balancer code (besides some other functionality that would be nice to
have). This trick would not work with clusters that use reweights. I
know it's a different discussion, but why do you use reweights? It
reminded me of this presentation I had seen at Cephalocon about the
"Echo" Ceph cluster (worth watching):
https://www.youtube.com/watch?v=niFNZN5EKvE
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
