On 08/23/2018 05:19 PM, Kefu Chai wrote: > + Mohamad > + ceph-devel > > ----- Original Message ----- >> From: "Yingxin Cheng" <yingxin.cheng@xxxxxxxxx> >> To: "Kefu Chai" <kchai@xxxxxxxxxx> >> Cc: "Chunmei Liu" <chunmei.liu@xxxxxxxxx>, "Jianpeng Ma" <jianpeng.ma@xxxxxxxxx> >> Sent: Wednesday, August 22, 2018 7:38:23 PM >> Subject: Thoughts about LTTng with seastar >> >> Hi Kefu, >> >> LTTng provides 3 modes [1]: discard, overwrite and blocking mode. >> “Overwrite” is another way to “discard”, so it is also not acceptable in Ceph >> logging I guess. >> >> After a second thought, I see “blocking mode” is reasonable because it only >> blocks when the sub-buffer is full [2]. >> IMO it would be contradictory if we want to write excessive logs with a >> limited buffer per core, and still don’t want blocking. > as seastar is using syslog(3) which in turn uses unix domain socket if it's directed > to a local logger. typically, syslogd reads from this socket, and write to /var/log. > so, it's also blocking if syslogd is not fast enough to read the data grams off from > the socket, and the sender is so fast that it fills up the socket send buffer (SO_SNDBUF). > the default setting of SO_SNDBUF is /proc/sys/net/core/wmem_max, which is 212992 = 208K > on my linux box. > > and the same applies to LTTng based logging, lttng-consumerd play the same role as > syslogd. as long as lttng-consumerd is able to consume the tracing events in the sub-buffer > fast enough, we are good. in an extreme case, we will block. but IIRC, seastar reactor > will be annoyed in that case. IIRC, it just complains if it's blocked for too long. > see reactor::block_notifier(). IMHO with a large enough buffer space, it'll be less common to run into these situations as the tracepoint size is quite small. Moreover, we can specify the number of subbuffers and their size with LTTng. For a fixed total buffer size, having more and smaller subbuffers reduces the probability of blocking/overwriting since a subbuffer can only be consumed when it's full and sealed. The downside of having more and smaller subbuffers is that sealing and switching subbuffers will happen more often than with few and larger subbufers, which is more costly. Mohamad > > i will give it a try to see what happens. > > >> And we will face the same problem when implement our own shared-nothing >> logging. >> >> The only concern now with LTTng would be its usability (also addressed by >> Gebai [3]) and extendibility (if we want some Ceph-specific modifications). >> And I really prefer simpler java/python-style logging interfaces. > please see https://github.com/ceph/ceph/pull/22458#discussion_r206821124 . i'd suggest > create a tool which works with lttng-view and our format definition file to > generate human-readable logging output. so, the > > ``` > trace("hello {}", who); > ``` > > style logging does not conflict with the LTTng backend. > >> [1] https://lttng.org/docs/v2.10/#doc-channel-overwrite-mode-vs-discard-mode >> [2] https://lttng.org/docs/v2.10/#doc-whats-new >> [3] https://www.spinics.net/lists/ceph-devel/msg41134.html >> >> -- >> Regards >> Yingxin >>
