Hi,
On 17.12.24 14:40, Janek Bevendorff wrote:
Hi all,
We moved our Ceph cluster to a new data centre about three months ago,
which completely changed its physical topology. I changed the CRUSH
map accordingly so that the CRUSH location matches the physical
location again and the cluster has been rebalancing ever since. Due to
capacity limits, the rebalancing requires rather frequent manual
reweights so that individual OSDs don't run full. We started at around
65% of remapped PGs and are now down to 12% (it goes a little faster
now that the bulk is done).
Unfortunately, the MONs refuse to trim their stores while there are
remapped PGs in the cluster, so their disk usage has increased
gradually from 600MB to 17GB. I assume the rebalancing will finish
before we run out of disk space, but the MON restart times have become
unsustainably long and compacts now take up to 20 minutes or so. I'm a
bit worried that at some point the MONs will start becoming unstable
or worse: the stores may get corrupted.
Is there anything that I can do to safely (!) get the MON stores back
to a more manageable size even without all PGs being active+clean? I
would like to forgo any potential disaster, especially before the
holidays.
Just my 0.02 euro...
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or
similar tools to cancel the remapping; up-map entries will be created
that reflect the current state of the cluster. After all currently
running backfills are finished your mons should not be blocked anymore.
I would also disable the balancer temporarily since it will trigger new
backfills for those PG that are not at their optimal locations. After
mons are fine again you can just enable the balancer. This requires a
ceph release and ceph clients with up-map support.
Not tested in real life, but this approach might work.
Best regards,
Burkhard
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
