On Tue, Jan 21, 2025 at 7:12 AM Anthony D'Atri <aad@xxxxxxxxxxxxxx> wrote: > > On Jan 21, 2025, at 7:59 AM, Kasper Rasmussen <kasper_steengaard@xxxxxxxxxxx> wrote: > > > > 1 - Why do this result in such a high - objects degraded - percentage? > > I suspect that’s a function of the new topology having changed the mappings of multiple OSDs for given PGs. It’s subtle, but when you move hosts into rack CRUSH buckets, that’s a different set of inputs into the CRUSH hash function, so the mappings that come out are different, even though you haven’t changed the rules and would think that hosts are hosts. Also, in the process of moving the hosts one by one, each step creates a new topology which can change the ordering of hosts, incrementally putting things out of whack. > > 2 - Why do PGs get undersized? > > That often means that CRUSH can’t find a complete set of placements. In your situation maybe those would resolve themselves when you unleash the recovery hounds. We started noticing this kind of issue around pacific, but haven't fully tracked down what broke yet. See https://tracker.ceph.com/issues/56046 for similar. Undersized or degraded should only happen -- by design -- if objects were modified while the PG did not have 3 OSDs up and acting. Kaspar: I assume the cluster was idle during your tests? Also -- can you reproduce it without norecover/nobackfill set ? Could you simplify your reproducer down to: > HEALTH_OK > ceph osd crush move ksr-ceph-osd1 rack=rack1 > ceph pg ls undersized / degraded # get a pgid of a degraded PG > ceph pg $pgid query Cheers, dan -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@xxxxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |