On 3/27/21 1:11 AM, Kefu Chai wrote:
hi folks,
i want to raise your attention to the tracker ticket of
https://tracker.ceph.com/issues/48909
<https://tracker.ceph.com/issues/48909>. and discuss with you for a
better solution.
some context first, back in https://github.com/ceph/ceph/pull/18614
<https://github.com/ceph/ceph/pull/18614>, changes were made so the slow
requests were reported to mgr to move the burden from monitor to mgr.
with that change, all health related reports are sent to mgr, and the
aggregated version is composed by mgr, and sent to monitor. i
think, that'd help to improve the scalability of a Ceph cluster.
moreover, IIUC, to let mgr take part of the load of the monitor was one
of the reasons why mgr was introduced in the first place.
in https://tracker.ceph.com/issues/43975
<https://tracker.ceph.com/issues/43975>, it's reported that the slow ops
were no longer recorded in cluster log anymore since mimic. as a fix,
https://github.com/ceph/ceph/pull/33328
<https://github.com/ceph/ceph/pull/33328> was created to send slow ops
and their types to cluster log.
in https://tracker.ceph.com/issues/43975
<https://tracker.ceph.com/issues/43975>, it's noticed that this fix even
worsen the performance of a cluster suffering from slow ops by adding
more load to monitor. hence https://github.com/ceph/ceph/pull/39199
<https://github.com/ceph/ceph/pull/39199> was created to throttle this.
i am wondering if we can make better use of the health reporting
machinery instead of pouring the health warnings to clog when slow ops
are observed?
what do you think?
Thanks for bringing this up Kefu, I agree there's a lot of room for
improvement here. It'd be a good topic for CDS.
There's no reason the cluster log needs to go through paxos or be stored
in the monitor DB, and some sort of throttling or data reduction would
help on the producer side. We've seen issues not just with slow ops
but other warnings reporting too frequently overloading the monitors
as well.
https://github.com/ceph/ceph/pull/40168 is related on the consumer side,
and also helps other cases of temporary mon overload (e.g. from a burst
of osdmap creation from blocklisting).
Josh
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
