+ Mohamad + ceph-devel // sorry for the reposting. the previous mail failed to reach ceph-devel ----- Original Message ----- > From: "Yingxin Cheng" <yingxin.cheng@xxxxxxxxx> > To: "Kefu Chai" <kchai@xxxxxxxxxx> > Cc: "Chunmei Liu" <chunmei.liu@xxxxxxxxx>, "Jianpeng Ma" <jianpeng.ma@xxxxxxxxx> > Sent: Wednesday, August 22, 2018 7:38:23 PM > Subject: Thoughts about LTTng with seastar > > Hi Kefu, > > LTTng provides 3 modes [1]: discard, overwrite and blocking mode. > “Overwrite” is another way to “discard”, so it is also not acceptable in Ceph > logging I guess. > > After a second thought, I see “blocking mode” is reasonable because it only > blocks when the sub-buffer is full [2]. > IMO it would be contradictory if we want to write excessive logs with a > limited buffer per core, and still don’t want blocking. as seastar is using syslog(3) which in turn uses unix domain socket if it's directed to a local logger. typically, syslogd reads from this socket, and write to /var/log. so, it's also blocking if syslogd is not fast enough to read the data grams off from the socket, and the sender is so fast that it fills up the socket send buffer (SO_SNDBUF). the default setting of SO_SNDBUF is /proc/sys/net/core/wmem_max, which is 212992 = 208K on my linux box. and the same applies to LTTng based logging, lttng-consumerd play the same role as syslogd. as long as lttng-consumerd is able to consume the tracing events in the sub-buffer fast enough, we are good. in an extreme case, we will block. but IIRC, seastar reactor will be annoyed in that case. IIRC, it just complains by printing out a backtrace if it's blocked for too long. see reactor::block_notifier(). > And we will face the same problem when implement our own shared-nothing > logging. > > The only concern now with LTTng would be its usability (also addressed by > Gebai [3]) and extendibility (if we want some Ceph-specific modifications). > And I really prefer simpler java/python-style logging interfaces. please see https://github.com/ceph/ceph/pull/22458#discussion_r206821124 . i'd suggest create a tool which works with lttng-view and our format definition file to generate human-readable logging output. so, the ``` trace("hello {}", who); ``` style logging does not conflict with the LTTng backend, if we support the "slow" logging proposed in "the logging in non-critical path" case. > > [1] https://lttng.org/docs/v2.10/#doc-channel-overwrite-mode-vs-discard-mode > [2] https://lttng.org/docs/v2.10/#doc-whats-new > [3] https://www.spinics.net/lists/ceph-devel/msg41134.html > > -- > Regards > Yingxin > -- Regards Kefu Chai
