On Wed, Apr 21, 2021 at 9:59 AM Kefu Chai <kchai@xxxxxxxxxx> wrote: > > hi folks, > > while looking at https://github.com/ceph/ceph/pull/32422, i think a probably safer approach is to make the monitor more efficient. currently, monitor is sort of a single-threaded application. quite a few critical code paths of monitor are protected by Monitor::lock, among other things > > - periodical task performed by tick() which is in turn called by SafeTimer. the "safty" of the SafeTimer is ensured by Monitor::lock > - Monitor::_ms_dispatch is also called with the Monitor::lock acquired. in the case of https://github.com/ceph/ceph/pull/32422, one or more kcephfs clients are even able to slow down the whole cluster by asking for the latest osdmap with an ancient one in its hand, if the cluster is able to rebalance/recover in speedy way and accumulate lots of osdmap in a short time. > > a typical scaring use case is: > > 1. an all-flash cluster just completes a rebalance/recover. the rebalance completed quickly, and it leaves the cluster with a ton of osdmaps before some of the clients have a chance to pick up these updated maps. > 2. (kcephfs) clients with ancient osdmaps in their hands wake up randomly, and they want the latest osdmap! > 3. monitors are occupied with loading the maps from rocksdb and encoding them in very large batches (when discussing with the author of https://github.com/ceph/ceph/pull/32422, he mentioned that the total size of inc osdmap could be up to 200~300 MiB). > 4. and the cluster is basically unresponsive. On the motivation side, also see the recent thread "Nautilus 14.2.19 mon 100% CPU" on ceph-users for a case where an old client was nearly 1M epochs behind and would never catch up so it continually loaded the mons. -- dan > > so, does it sound like a right way to improve its performance when serving the CPU intensive workload by dissecting the data dependencies in the monitor and to explore the possibility to make the monitor more multi-threaded? > > thoughts? > _______________________________________________ > Dev mailing list -- dev@xxxxxxx > To unsubscribe send an email to dev-leave@xxxxxxx _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
