Update: we have run fsck and re-shard on all bluestore volume, seems sharding were not applied. Unfortunately scrubs and deep-scrubs are still stuck on PGs of the pool that is suffering the issue, but other PGs scrubs well. The next step will be to remove the cache tier as suggested, but its not available yet as PGs needs to be scrubbed in order for the cache tier can be activated. As we are struggling to make this cluster works again, any help would be greatly appreciated. Cédric > On 20 Feb 2024, at 20:22, Cedric <yipikai7@xxxxxxxxx> wrote: > > Thanks Eugen, sorry about the missed reply to all. > > The reason we still have the cache tier is because we were not able to flush all dirty entry to remove it (as per the procedure), so the cluster as been migrated from HDD/SSD to NVME a while ago but tiering remains, unfortunately. > > So actually we are trying to understand the root cause > > On Tue, Feb 20, 2024 at 1:43 PM Eugen Block <eblock@xxxxxx> wrote: >> >> Please don't drop the list from your response. >> >> The first question coming to mind is, why do you have a cache-tier if >> all your pools are on nvme decices anyway? I don't see any benefit here. >> Did you try the suggested workaround and disable the cache-tier? >> >> Zitat von Cedric <yipikai7@xxxxxxxxx>: >> >>> Thanks Eugen, see attached infos. >>> >>> Some more details: >>> >>> - commands that actually hangs: ceph balancer status ; rbd -p vms ls ; >>> rados -p vms_cache cache-flush-evict-all >>> - all scrub running on vms_caches pgs are stall / start in a loop >>> without actually doing anything >>> - all io are 0 both from ceph status or iostat on nodes >>> >>> On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock@xxxxxx> wrote: >>>> >>>> Hi, >>>> >>>> some more details would be helpful, for example what's the pool size >>>> of the cache pool? Did you issue a PG split before or during the >>>> upgrade? This thread [1] deals with the same problem, the described >>>> workaround was to set hit_set_count to 0 and disable the cache layer >>>> until that is resolved. Afterwards you could enable the cache layer >>>> again. But keep in mind that the code for cache tier is entirely >>>> removed in Reef (IIRC). >>>> >>>> Regards, >>>> Eugen >>>> >>>> [1] >>>> https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-adding-a-cache-osd >>>> >>>> Zitat von Cedric <yipikai7@xxxxxxxxx>: >>>> >>>>> Hello, >>>>> >>>>> Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we >>>>> encounter an issue with a cache pool becoming completely stuck, >>>>> relevant messages below: >>>>> >>>>> pg xx.x has invalid (post-split) stats; must scrub before tier agent >>>>> can activate >>>>> >>>>> In OSD logs, scrubs are starting in a loop without succeeding for all >>>>> pg of this pool. >>>>> >>>>> What we already tried without luck so far: >>>>> >>>>> - shutdown / restart OSD >>>>> - rebalance pg between OSD >>>>> - raise the memory on OSD >>>>> - repeer PG >>>>> >>>>> Any idea what is causing this? any help will be greatly appreciated >>>>> >>>>> Thanks >>>>> >>>>> Cédric >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |