I found a config to force scrub invalid PGs, what is your current
setting on that?
ceph config get osd osd_scrub_invalid_stats
true
The config reference states:
Forces extra scrub to fix stats marked as invalid.
But the default seems to be true, so I'd expect it's true in your case
as well?
Zitat von Cedric <yipikai7@xxxxxxxxx>:
Thanks Eugen for the suggestion, yes we have tried, also repeering
concerned PGs, still the same issue.
Looking at the code it seems the split-mode message is triggered when
the PG as ""stats_invalid": true,", here is the result of a query:
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
I also provide again cluster informations that was lost in previous
missed reply all. Don't hesitate to ask more if needed I would be
glade to provide them.
Cédric
On Thu, Feb 22, 2024 at 11:04 AM Eugen Block <eblock@xxxxxx> wrote:
Hm, I wonder if setting (and unsetting after a while) noscrub and
nodeep-scrub has any effect. Have you tried that?
Zitat von Cedric <yipikai7@xxxxxxxxx>:
> Update: we have run fsck and re-shard on all bluestore volume, seems
> sharding were not applied.
>
> Unfortunately scrubs and deep-scrubs are still stuck on PGs of the
> pool that is suffering the issue, but other PGs scrubs well.
>
> The next step will be to remove the cache tier as suggested, but its
> not available yet as PGs needs to be scrubbed in order for the cache
> tier can be activated.
>
> As we are struggling to make this cluster works again, any help
> would be greatly appreciated.
>
> Cédric
>
>> On 20 Feb 2024, at 20:22, Cedric <yipikai7@xxxxxxxxx> wrote:
>>
>> Thanks Eugen, sorry about the missed reply to all.
>>
>> The reason we still have the cache tier is because we were not able
>> to flush all dirty entry to remove it (as per the procedure), so
>> the cluster as been migrated from HDD/SSD to NVME a while ago but
>> tiering remains, unfortunately.
>>
>> So actually we are trying to understand the root cause
>>
>> On Tue, Feb 20, 2024 at 1:43 PM Eugen Block <eblock@xxxxxx> wrote:
>>>
>>> Please don't drop the list from your response.
>>>
>>> The first question coming to mind is, why do you have a cache-tier if
>>> all your pools are on nvme decices anyway? I don't see any benefit here.
>>> Did you try the suggested workaround and disable the cache-tier?
>>>
>>> Zitat von Cedric <yipikai7@xxxxxxxxx>:
>>>
>>>> Thanks Eugen, see attached infos.
>>>>
>>>> Some more details:
>>>>
>>>> - commands that actually hangs: ceph balancer status ; rbd -p vms ls ;
>>>> rados -p vms_cache cache-flush-evict-all
>>>> - all scrub running on vms_caches pgs are stall / start in a loop
>>>> without actually doing anything
>>>> - all io are 0 both from ceph status or iostat on nodes
>>>>
>>>> On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock@xxxxxx> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> some more details would be helpful, for example what's the pool size
>>>>> of the cache pool? Did you issue a PG split before or during the
>>>>> upgrade? This thread [1] deals with the same problem, the described
>>>>> workaround was to set hit_set_count to 0 and disable the cache layer
>>>>> until that is resolved. Afterwards you could enable the cache layer
>>>>> again. But keep in mind that the code for cache tier is entirely
>>>>> removed in Reef (IIRC).
>>>>>
>>>>> Regards,
>>>>> Eugen
>>>>>
>>>>> [1]
>>>>>
https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-adding-a-cache-osd
>>>>>
>>>>> Zitat von Cedric <yipikai7@xxxxxxxxx>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we
>>>>>> encounter an issue with a cache pool becoming completely stuck,
>>>>>> relevant messages below:
>>>>>>
>>>>>> pg xx.x has invalid (post-split) stats; must scrub before tier agent
>>>>>> can activate
>>>>>>
>>>>>> In OSD logs, scrubs are starting in a loop without succeeding for all
>>>>>> pg of this pool.
>>>>>>
>>>>>> What we already tried without luck so far:
>>>>>>
>>>>>> - shutdown / restart OSD
>>>>>> - rebalance pg between OSD
>>>>>> - raise the memory on OSD
>>>>>> - repeer PG
>>>>>>
>>>>>> Any idea what is causing this? any help will be greatly appreciated
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Cédric
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>
>>>
>>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
