Last week there was a great talk at Open Source Summit on how opentelemetry (OTEL) is used by the nginx team. The video is not up yet, but the slides are here [1].
A couple of takeaways from that talk:
* tracing: they selected the OTEL + jaeger combination for tracing :-)
* logging: they don't use OTEL for logging as this is not mature yet (they have a different stack, see slide 19). however, they added the "trace-id" to their logging system, as it is proved to be valuable information when correlating traces and logs
* we may consider a similar mechanism by adding an optional trace [2] to the DoutPrefixProvider interface [3]
* converging logging and tracing together (using the "AddEvent" OTEL API) was not recommended, as tracing is often sampled
* metrics: according to the talk, OTEL has a mature solution for metrics.
* one option is to use the SDK in order to push metrics outside of our demons, by using exporters [4]
* the other option, which is similar to how our newly developed exporter [5] works, is to extend the OTEL collector [6] (which is written in golang) so it would scrape the metrics from our daemons
* libraries: their main challenge was to integrate OTEL with the Java frameworks they used. Well, we don't have this problem... we are tracing our core flows where most of the code is ours.
unrelated to that specific talk, but since nginx uses OTEL for its own traces [7], it might be useful to integrate into our jaeger backend when we use nginx for RGW caching [8]
appreciate your feedback,
Yuval
[1] https://static.sched.com/hosted_files/ossna2022/17/LF%20CloudOpen%20Lessons%20Learned%20OTel.pdf
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
