I think it was mentioned elsewhere in this thread that there are limitations to what upmap can do, especially in significant crush map change situations. It can't violate crush rules (mon-enforced), and if the same OSD shows up multiple times in a backfill then upmap can't deal with it.
The number of warnings was larger the first few times I ran it, but I think this is as low as I can get for now. I also just replaced one HDD with excessive read/write times, which neared the end of its lifespan and kept flip-flopping. So for now I'm waiting for that to backfill again, then I'll try again tomorrow.
Creeping back up is a bit odd; if you have the balancer off, any chance there's somehow also a PG split going on? What does 'ceph osd pool ls detail' say?
Indeed! I did increase the pgp_num of a pool a while back, totally forgot about that. Due to the ongoing rebalancing it was stuck half way, but now suddenly started up again. The current PG number of that pool is not quite final yet, but definitely higher than previously.
I'll keep this running over night and see where it settles. Thanks so far! Janek
Josh On Tue, Dec 17, 2024 at 10:06 AM Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote:Something's not quite right yet. I got the remapped PGs down from > 4000 to around 1300, but there it stops. When I restart the process, I can get it down to around 280, but there it stops and creeps back up afterwards. I have a bunch of these messages in the output: WARNING: pg 100.3d53: conflicting mapping 1068->1051 found when trying to map 187->1068 There's maybe around 70-80 of them (definitely not 280 or 1300), any idea how I can fix that? The messages all point to the same pool (our largest one, I did not change the failure domain for this pool).Ah, yes, we ran into that invalid json output in https://github.com/digitalocean/ceph_exporter as well. I have a patch I wrote for ceph_exporter that I can port over to pgremapper (that does similar to what your patch does).That'd be nice! Janek
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
