Hi Yingxin, This is great, and timely. We definitely want to sort out what the "right" way to structure tracepoints is going forward. Perhaps a useful initial step is to add some basic tracepoints for something simple that demonstrate the best practice. Maybe messenger sends/receives? Or some parts of the data path? Josh suggested last week that instead of bothering to reimplement an efficient dout mechanism for seastar we instead focus on using structured tracepoints in the reimplemented code. On Fri, 13 Apr 2018, Yingxin Cheng wrote: > Hi, > > Tracing in Ceph is very complicated with lttng tracepoints, function > traces, babeltraces, op trackers and blkin... They are hard to use > because most of them lack essential features such as cross-component > tracing, sampling, trace data centralization and analysis pipeline, > all requiring considerable efforts to implement. > > OpenTracing (opentracing.io) addresses these problems by providing > vendor-agnostic APIs for distributed tracing. It means that by > leveraging pure OpenTracing APIs, essential features can be available > immediately with compatible tracers. For example, Jaeger tracer > (www.jaegertracing.io) already supports trace context propagation, > adaptive sampling with remote control, trace data > buffering/report/persist and simple web UI. And with O(1) > configuration change, other trace plugins can be used for unittest and > more advanced purposes. Most importantly, efforts can be divided to > implement Ceph-native tracer plugin, bringing Ceph-specific analysis > and interoperable features in dashboard/mgr/central-config, etc., > without dependencies on 3rd-party trace systems. > > I wrote a simplest trace library > (https://github.com/cyx1231st/distributed-tracing-example) with > jaegerTracer/mockTracer plugin and Ceph simulator as a start. I’m > looking forward to implement Ceph tracing library based on this if > necessary. I keep getting fuzzy on this point. My understanding is that there isn't really a "opentracing library," but that existing structured traces of whatever form (including LTTNG traces) can be injested. Is that right? Is there anything we really need to do on this front beyond documenting and adopting a set of best practices around how the tracepoints are declared and placed in the code? Thanks! sage > I’m also working on extending the OpenTracing model to support > “CriticalPath-aware tracing”, as mentioned in Ceph Alocon Session > “Pinpoint Ceph Bottleneck Out of Cluster Behavior Mists”. It is a > solution for cluster-level bottleneck identification & data-driven > analysis. > > > Regards > Yingxin > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > >
