Hi Dominic, All, After going through the errors in detail and looking through "ceph features", I have set *ceph osd set-require-min-compat-client luminous* and cleared the warning. I have fixed the remaining warning too and the cluster is healthy. Thank you everyone for taking time to respond. Regards, suresh On Fri, Jul 9, 2021 at 3:37 PM <DHilsbos@xxxxxxxxxxxxxx> wrote: > Suresh; > > I don't believe we use tunables, so I'm not terribly familiar with them. > > A quick Google search ("ceph tunable") supplied the following pages: > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/html/storage_strategies/crush_tunables > https://docs.ceph.com/en/latest/rados/operations/crush-map/#tunables > > Thank you, > > Dominic L. Hilsbos, MBA > Vice President - Information Technology > Perform Air International Inc. > DHilsbos@xxxxxxxxxxxxxx > www.PerformAir.com > > -----Original Message----- > From: Suresh Rama [mailto:sstkadu@xxxxxxxxx] > Sent: Thursday, July 8, 2021 7:25 PM > To: ceph-users > Subject: Issue with Nautilus upgrade from Luminous > > Dear All, > > We have 13 Ceph clusters and we started upgrading one by one from Luminous > to Nautilus. Post upgrade started fixing the warning alerts and had issues > setting "*ceph config set mon mon_crush_min_required_version firefly" > *yielded > no results. Updated the mon config and restart the daemons the warning > didn't go away > > I have also tried to set it to hammer and no use. The warning is still > there. Do you have any recommendations? I thought of changing it to > hammer so I can use straw2 but I was stuck with warning message. I have > also bounced the nodes and the issue remains the same. > > Please review and share your inputs. > > cluster: > id: xxxxxxxxxxx > health: HEALTH_WARN > crush map has legacy tunables (require firefly, min is hammer) > 1 pools have many more objects per pg than average > 15252 pgs not deep-scrubbed in time > 21399 pgs not scrubbed in time > clients are using insecure global_id reclaim > mons are allowing insecure global_id reclaim > 3 monitors have not enabled msgr2 > > > ceph daemon mon.$(hostname -s) config show |grep -i > mon_crush_min_required_version > "mon_crush_min_required_version": "firefly", > > ceph osd crush show-tunables > { > "choose_local_tries": 0, > "choose_local_fallback_tries": 0, > "choose_total_tries": 50, > "chooseleaf_descend_once": 1, > "chooseleaf_vary_r": 1, > "chooseleaf_stable": 0, > "straw_calc_version": 1, > "allowed_bucket_algs": 22, > "profile": "firefly", > "optimal_tunables": 0, > "legacy_tunables": 0, > "minimum_required_version": "firefly", > "require_feature_tunables": 1, > "require_feature_tunables2": 1, > "has_v2_rules": 0, > "require_feature_tunables3": 1, > "has_v3_rules": 0, > "has_v4_buckets": 0, > "require_feature_tunables5": 0, > "has_v5_rules": 0 > } > > ceph config dump > WHO MASK LEVEL OPTION VALUE RO > mon advanced mon_crush_min_required_version firefly * > > ceph versions > { > "mon": { > "ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > nautilus (stable)": 3 > }, > "mgr": { > "ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > nautilus (stable)": 3 > }, > "osd": { > "ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) > nautilus (stable)": 549, > "ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > nautilus (stable)": 226 > }, > "mds": {}, > "rgw": { > "ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > nautilus (stable)": 2 > }, > "overall": { > "ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) > nautilus (stable)": 549, > "ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) > nautilus (stable)": 234 > } > } > > ceph -s > cluster: > id: xxxxxxxxxxxxxxxxxx > health: HEALTH_WARN > crush map has legacy tunables (require firefly, min is hammer) > 1 pools have many more objects per pg than average > 13811 pgs not deep-scrubbed in time > 19994 pgs not scrubbed in time > clients are using insecure global_id reclaim > mons are allowing insecure global_id reclaim > 3 monitors have not enabled msgr2 > > services: > mon: 3 daemons, quorum > pistoremon-ho-c01,pistoremon-ho-c02,pistoremon-ho-c03 (age 24s) > mgr: pistoremon-ho-c02(active, since 2m), standbys: pistoremon-ho-c01, > pistoremon-ho-c03 > osd: 800 osds: 775 up (since 105m), 775 in > rgw: 2 daemons active (pistorergw-ho-c01, pistorergw-ho-c02) > > task status: > > data: > pools: 28 pools, 27336 pgs > objects: 107.19M objects, 428 TiB > usage: 1.3 PiB used, 1.5 PiB / 2.8 PiB avail > pgs: 27177 active+clean > 142 active+clean+scrubbing+deep > 17 active+clean+scrubbing > > io: > client: 220 MiB/s rd, 1.9 GiB/s wr, 7.07k op/s rd, 25.42k op/s wr > > -- > Regards, > Suresh > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > -- Regards, Suresh _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx