Hi Sage, >From my experience, trace the highest-level datapath of the read/write operations from RADOS to OSDs can be a good start. This will contain most of the logical scenarios (cross-component/queues/mutex/forks/joins) which force us to refine the tracing model for Ceph at the very beginning. Also, this helps to measure the actual tracing overhead and to consider how to get useful cluster-level results leveraging tracing data. About the "opentracing library" in Ceph, a simple illustration here: +-----------------------------------+ | Ceph Code | +-----------------------------------+ | Ceph tracing library | | (syntactic sugars/plugin control) | | (improved tracing model) | +-----------------------------------+ | Pure OpenTracing versioned API | +-----------------------------------+ | Tracer plugins | | (Jaeger/Zipkin/LTTNG/mock/dout) | +-----------------------------------+ >From bottom to top: The "structured traces" is actually handled by Tracer plugins. To generate traces by LTTNG/dout, new plugin needs to be implemented. It will look like the current Blkin library, but hidden under OpenTracing API. Important features such as sampling, trace-data centralization & persistence still need efforts to implement. By switching to Jaeger tracer plugin (see demo [1]), those features are ready immediately. However, jaeger has its own implementation to process traces [2]. The performance penalty is unknown, and Jaeger also lacks interoperability with Ceph in the long term. OpenTracing provides consistent APIs for the traced system like Ceph. So that Ceph tracepoints can be decoupled with tracer implementations [3]. This means we can use mocked tracer [4] for testing, Jaeger tracer [1] for immediate features, and Ceph-specific tracer for performance considerations, project-specific analysis and future interoperability. Ceph tracing library is needed because the direct usage of OpenTracing API is too verbose [5]. Syntactic sugars are good to make things simpler. Some Ceph-specific configurations and plugin control need be integrated. And OpenTracing model can be improved to be "critical-path-aware" leveraging span tags and logs [6]. Tags and logs can also be encapsulated to generate high-level Ceph performance metrics. <ref> [1] Jaeger tracer: https://github.com/cyx1231st/distributed-tracing-example/blob/master/src/main.cc#L500 [2] Jaeger tracer reporter: https://www.jaegertracing.io/docs/client-libraries/#reporters [3] The traced code only depends on OpenTracing API: https://github.com/cyx1231st/distributed-tracing-example/blob/master/src/main.cc#L15 [4] Mock tracer: https://github.com/cyx1231st/distributed-tracing-example/blob/master/src/main.cc#L498 [5] OpenTracing API usage: https://github.com/cyx1231st/distributed-tracing-example/blob/master/src/main.cc#L274-L283 #L289-L294 #L298 [6] Tag and Log: https://github.com/cyx1231st/distributed-tracing-example/blob/master/src/main.cc#L529 #525 Regards, Yingxin 2018-04-16 20:18 GMT+08:00 Sage Weil <sage@xxxxxxxxxxxx>: > Hi Yingxin, > > This is great, and timely. We definitely want to sort out what the > "right" way to structure tracepoints is going forward. Perhaps a useful > initial step is to add some basic tracepoints for something simple that > demonstrate the best practice. Maybe messenger sends/receives? Or some > parts of the data path? > > Josh suggested last week that instead of bothering to reimplement an > efficient dout mechanism for seastar we instead focus on using structured > tracepoints in the reimplemented code. > > On Fri, 13 Apr 2018, Yingxin Cheng wrote: >> Hi, >> >> Tracing in Ceph is very complicated with lttng tracepoints, function >> traces, babeltraces, op trackers and blkin... They are hard to use >> because most of them lack essential features such as cross-component >> tracing, sampling, trace data centralization and analysis pipeline, >> all requiring considerable efforts to implement. >> >> OpenTracing (opentracing.io) addresses these problems by providing >> vendor-agnostic APIs for distributed tracing. It means that by >> leveraging pure OpenTracing APIs, essential features can be available >> immediately with compatible tracers. For example, Jaeger tracer >> (www.jaegertracing.io) already supports trace context propagation, >> adaptive sampling with remote control, trace data >> buffering/report/persist and simple web UI. And with O(1) >> configuration change, other trace plugins can be used for unittest and >> more advanced purposes. Most importantly, efforts can be divided to >> implement Ceph-native tracer plugin, bringing Ceph-specific analysis >> and interoperable features in dashboard/mgr/central-config, etc., >> without dependencies on 3rd-party trace systems. >> >> I wrote a simplest trace library >> (https://github.com/cyx1231st/distributed-tracing-example) with >> jaegerTracer/mockTracer plugin and Ceph simulator as a start. I’m >> looking forward to implement Ceph tracing library based on this if >> necessary. > > I keep getting fuzzy on this point. My understanding is that there isn't > really a "opentracing library," but that existing structured traces > of whatever form (including LTTNG traces) can be injested. Is that right? > Is there anything we really need to do on this front beyond documenting > and adopting a set of best practices around how the tracepoints are > declared and placed in the code? > > Thanks! > sage > > >> I’m also working on extending the OpenTracing model to support >> “CriticalPath-aware tracing”, as mentioned in Ceph Alocon Session >> “Pinpoint Ceph Bottleneck Out of Cluster Behavior Mists”. It is a >> solution for cluster-level bottleneck identification & data-driven >> analysis. >> >> >> Regards >> Yingxin >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
