So in other words - it's unsafe to apply quick-fix/repair in 16.2.6
only. You're safe if you applied it before or newly deployed OSDs with
v16 (even 16.2.6).
Igor
On 1/20/2022 5:22 PM, Jay Sullivan wrote:
I just verified the following in my other 16.2.6 cluster:
(S, per_pool_omap)
00000000 32 |2|
00000001
I set noout, stopped the OSD service, ran the "ceph-kvstore-tool bluestore-kv <path-to-osd> get S per_pool_omap" command, and started the OSD back up.
Looking back through my change history, I had manually set bluestore_fsck_quick_fix_on_mount to "true" right before my upgrade from Nautilus 14.2.20 to Pacific 16.2.4. Per the "holy war" (lol) referenced below and elsewhere, I figured I'd just get the OMAP conversion out of the way while I was already in a maintenance window:
https://tracker.ceph.com/issues/45265
My OSDs are not crashing on startup. I would have seen something by now, right? How was I spared? This specific cluster is mostly RBD (79M objects for 295TB data) with a little RGW (65k objects for 200GB data) and a very small non-production CephFS volume (1GB).
~Jay
-----Original Message-----
From: Jay Sullivan
Sent: Wednesday, January 19, 2022 12:36 PM
To: 'Igor Fedotov' <igor.fedotov@xxxxxxxx>; ceph-users@xxxxxxx
Subject: RE: Re: Scope of Pacific 16.2.6 OMAP Keys Bug?
Hello, Igor!
I just ran "ceph config set osd bluestore_warn_on_no_per_pg_omap true". No warnings yet. Would I need to restart any services for that to take effect?
When I upgraded from Nautilus to Pacific 16.2.4 in July and then 16.2.4 to 16.2.6 in mid-October I upgraded the OSD packages, restarted the OSD services, waited for HEALTH_OK, and then rebooted the OSD hosts one at a time. I've had a handful of drive failures since installing 16.2.6 so brand new OSDs would have started up with bluestore_fsck_quick_fix_on_mount set to "true" (not sure if that counts). But to answer your question, all of my OSDs have restarted at least once since upgrading to 16.2.6 in the form of a full host reboot. I have not seen any unplanned OSD restarts.
I just checked one of my 16.2.6 clusters and I see the following (I think this particular cluster was "born" on Nautilus). I'll check my other 16.2.6 cluster in a bit as it's currently backfilling from a drive failure earlier this morning.
(S, per_pool_omap)
00000000 32 |2|
00000001
Would the per-pg OMAP task have run during my upgrade from Nautilus/Octopus to Pacific 16.2.4? (I think YES.) Was the bug introduced specifically in 16.2.6 or was it present in older versions, too? Am I possibly spared because I upgraded to 16.2.4 first before the OMAP bug was introduced? Again, I have not seen any unplanned OSD restarts.
Thanks, Igor!!
~Jay
-----Original Message-----
From: Igor Fedotov <igor.fedotov@xxxxxxxx>
Sent: Wednesday, January 19, 2022 8:24 AM
To: Jay Sullivan <jpspgd@xxxxxxx>; ceph-users@xxxxxxx
Subject: Re: Scope of Pacific 16.2.6 OMAP Keys Bug?
Hey Jay,
first of all I'd like to mention that there were two OMAP naming scheme modifications since Nautilus:
1) per-pool OMAPs
2) per-pg OMAPs
Both are applied during BlueStore repair/quick-fix. So may be you performed the first one but not the second.
You might want to set bluestore_warn_on_no_per_pg_omap to true and inspect ceph health alerts to learn if per-pg format is disabled in your cluster.
Alternatively you might want to inspect db manually against an offline OSD with ceph-kvstore-tool: ceph-kvstore-tool bluestore-kv <path-to-osd> get S per_pool_omap:
(S, per_pool_omap)
00000000 32 |2|
00000001
Having ASCII '2' there means per-pg format is already applied. I can't explain why you're observing no issues if that's the case though...
As for your other questions - you can stay on 16.2.6 as far as you don't run BlueStore repair/quick-fix - i.e. the relevant setting is false and nobody runs relevant ceph-bluestore-tool commands manually.
And you mentioned bluestore_fsck_quick_fix_on_mount was set to true for until now - curious if you had any OSD restarts with that setting set to true?
Thanks,
Igor
On 1/19/2022 4:28 AM, Jay Sullivan wrote:
https://tracker.ceph.com/issues/53062
Can someone help me understand the scope of the OMAP key bug linked above? I’ve been using 16.2.6 for three months and I don’t _think_ I’ve seen any related problems.
I upgraded my Nautilus (then 14.2.21) clusters to Pacific (16.2.4) in mid-June. One of my clusters was born in Jewel and has made all of the even-numbered releases to Pacific. I skipped over 16.2.5 and upgraded to 16.2.6 in mid-October. It looks like the aforementioned OMAP bug was discovered shortly after, on/around October 20th. My clusters had bluestore_fsck_quick_fix_on_mount set to true until about 10 minutes ago. I _think_ all of my OSDs did the OMAP conversion when I upgraded from Nautilus to Pacific back in June (I remember it taking several minutes per spinning OSD).
Questions for my sanity:
* Do I need to upgrade to 16.2.7 ASAP? Or can I wait until my next regular maintenance window?
* What is the risk of staying on 16.2.6 if I have bluestore_fsck_quick_fix_on_mount set to false?
* If I don’t have OSDs crashing, how would I know if I was impacted by the bug?
Thanks! ❤
~Jay
--
Jay Sullivan
Rochester Institute of Technology
jay.sullivan@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
