I was given a cluster recently where they'd managed to enable upmap despite clients not supporting it, and then on removing the upmap features the clients were triggering monitor crashes. While trying to establish what had happened, I discovered that most of the controls in this area have become pretty stale or broken! So far as I can tell: * Admins can invoke "osd set -require-min-compat-client", and unless you override it tries check against already-connected clients * but the check makes use of ceph_release_features() (which I've recently noted to be a bit weak) and also only checks against the entities which are connected to the local monitor making the change. * As best I can tell, setting a min_compat_client does *not* actually require newly-connected entities to have the required feature set! * The OSDMonitor DOES require any connecting entity satisfy the features demanded by OSDMap::get_features() (via update_msgr_features(), called from OSDMonitor::update_from_paxos()) * but this function has not been reliably updated? It adds CEPH_FEATUREMASK_SERVER_KRAKEN for OSDs (but not others!) and is configured to add CEPH_FEATUREMASK_CEPHX_V2 when appropriate, but has nothing from luminous (like, say...UPMAP). Am I misunderstanding something here? Am I correct in thinking that we just need to wire up require_min_compat_client to OSDMap::get_features(), or is there another enforcement mechanism we're looking for? -Greg
