You moved some oSD’s I believe it’s looking for peer osds data too. But as long as you keep nobackfill,norebalance,norecover it will take longer and keep showing more data to balance, and kept filling or writing data to volumes it will accumulate. So unset and wait for some time to finish it. Regards Dev On Tue, 21 Jan 2025 at 7:53 AM, Dan van der Ster <dan.vanderster@xxxxxxxxx> wrote: > On Tue, Jan 21, 2025 at 7:12 AM Anthony D'Atri <aad@xxxxxxxxxxxxxx> wrote: > > > On Jan 21, 2025, at 7:59 AM, Kasper Rasmussen < > kasper_steengaard@xxxxxxxxxxx> wrote: > > > > > > 1 - Why do this result in such a high - objects degraded - percentage? > > > > I suspect that’s a function of the new topology having changed the > mappings of multiple OSDs for given PGs. It’s subtle, but when you move > hosts into rack CRUSH buckets, that’s a different set of inputs into the > CRUSH hash function, so the mappings that come out are different, even > though you haven’t changed the rules and would think that hosts are hosts. > > Also, in the process of moving the hosts one by one, each step creates > a new topology which can change the ordering of hosts, incrementally > putting things out of whack. > > > > 2 - Why do PGs get undersized? > > > > That often means that CRUSH can’t find a complete set of placements. In > your situation maybe those would resolve themselves when you unleash the > recovery hounds. > > We started noticing this kind of issue around pacific, but haven't > fully tracked down what broke yet. > See https://tracker.ceph.com/issues/56046 for similar. > > Undersized or degraded should only happen -- by design -- if objects > were modified while the PG did not have 3 OSDs up and acting. > Kaspar: I assume the cluster was idle during your tests? > Also -- can you reproduce it without norecover/nobackfill set ? > > Could you simplify your reproducer down to: > > > HEALTH_OK > > ceph osd crush move ksr-ceph-osd1 rack=rack1 > > ceph pg ls undersized / degraded # get a pgid of a degraded PG > > ceph pg $pgid query > > Cheers, dan > > > -- > Dan van der Ster > CTO @ CLYSO > Try our Ceph Analyzer -- https://analyzer.clyso.com/ > https://clyso.com | dan.vanderster@xxxxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |