Hello, Igor! I just ran "ceph config set osd bluestore_warn_on_no_per_pg_omap true". No warnings yet. Would I need to restart any services for that to take effect? When I upgraded from Nautilus to Pacific 16.2.4 in July and then 16.2.4 to 16.2.6 in mid-October I upgraded the OSD packages, restarted the OSD services, waited for HEALTH_OK, and then rebooted the OSD hosts one at a time. I've had a handful of drive failures since installing 16.2.6 so brand new OSDs would have started up with bluestore_fsck_quick_fix_on_mount set to "true" (not sure if that counts). But to answer your question, all of my OSDs have restarted at least once since upgrading to 16.2.6 in the form of a full host reboot. I have not seen any unplanned OSD restarts. I just checked one of my 16.2.6 clusters and I see the following (I think this particular cluster was "born" on Nautilus). I'll check my other 16.2.6 cluster in a bit as it's currently backfilling from a drive failure earlier this morning. (S, per_pool_omap) 00000000 32 |2| 00000001 Would the per-pg OMAP task have run during my upgrade from Nautilus/Octopus to Pacific 16.2.4? (I think YES.) Was the bug introduced specifically in 16.2.6 or was it present in older versions, too? Am I possibly spared because I upgraded to 16.2.4 first before the OMAP bug was introduced? Again, I have not seen any unplanned OSD restarts. Thanks, Igor!! ~Jay -----Original Message----- From: Igor Fedotov <igor.fedotov@xxxxxxxx> Sent: Wednesday, January 19, 2022 8:24 AM To: Jay Sullivan <jpspgd@xxxxxxx>; ceph-users@xxxxxxx Subject: Re: Scope of Pacific 16.2.6 OMAP Keys Bug? Hey Jay, first of all I'd like to mention that there were two OMAP naming scheme modifications since Nautilus: 1) per-pool OMAPs 2) per-pg OMAPs Both are applied during BlueStore repair/quick-fix. So may be you performed the first one but not the second. You might want to set bluestore_warn_on_no_per_pg_omap to true and inspect ceph health alerts to learn if per-pg format is disabled in your cluster. Alternatively you might want to inspect db manually against an offline OSD with ceph-kvstore-tool: ceph-kvstore-tool bluestore-kv <path-to-osd> get S per_pool_omap: (S, per_pool_omap) 00000000 32 |2| 00000001 Having ASCII '2' there means per-pg format is already applied. I can't explain why you're observing no issues if that's the case though... As for your other questions - you can stay on 16.2.6 as far as you don't run BlueStore repair/quick-fix - i.e. the relevant setting is false and nobody runs relevant ceph-bluestore-tool commands manually. And you mentioned bluestore_fsck_quick_fix_on_mount was set to true for until now - curious if you had any OSD restarts with that setting set to true? Thanks, Igor On 1/19/2022 4:28 AM, Jay Sullivan wrote: > https://tracker.ceph.com/issues/53062 > > Can someone help me understand the scope of the OMAP key bug linked above? I’ve been using 16.2.6 for three months and I don’t _think_ I’ve seen any related problems. > > I upgraded my Nautilus (then 14.2.21) clusters to Pacific (16.2.4) in mid-June. One of my clusters was born in Jewel and has made all of the even-numbered releases to Pacific. I skipped over 16.2.5 and upgraded to 16.2.6 in mid-October. It looks like the aforementioned OMAP bug was discovered shortly after, on/around October 20th. My clusters had bluestore_fsck_quick_fix_on_mount set to true until about 10 minutes ago. I _think_ all of my OSDs did the OMAP conversion when I upgraded from Nautilus to Pacific back in June (I remember it taking several minutes per spinning OSD). > > Questions for my sanity: > > * Do I need to upgrade to 16.2.7 ASAP? Or can I wait until my next regular maintenance window? > * What is the risk of staying on 16.2.6 if I have bluestore_fsck_quick_fix_on_mount set to false? > * If I don’t have OSDs crashing, how would I know if I was impacted by the bug? > > Thanks! ❤ > > ~Jay > > -- > Jay Sullivan > Rochester Institute of Technology > jay.sullivan@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |