On Tue, 2 May 2017, Mohamad Gebai wrote: > Hi, > > I thought I would share some results of tracing I've been conducting recently. > I instrumented Ceph (OSD only) with GCC's -finstrument-functions option, and > traced it and the kernel simultaneously using LTTng (no instrumentation is > actually necessary, though rebuilding Ceph is required). > > In [1]: > - the top view shows the state of the OSD process (22983). It is switching > between the running (green) and blocked (yellow) states > - the middle view shows the critical path of that process, in other words it > shows what the process 22983 was waiting for when it is in the blocked > (yellow) state > - the bottom view shows the call stack graph, which is all function calls in > OSD code, generated by -finstrument-functions > > Image [2] shows a better image of the call stack view, and Image [3] shows a > better one of the critical path. > > The code is in branch [4]. These are awesome! What is the tool you're screenshotting to visualize? It would be great to apply the build options based on an option passed to cmake (e.g, -DWITH_INSTRUMENT_FUNCTIONS). Also, I wonder if these are a better/simpler alternative to the FUNCTRACE() macro in src/common/EventTrace.h. What kind of overhead is introduced? LTTNG lets you enable on specific tracepoints, right? > After looking extensively at the result, I created a diagram that simplifies > these interactions and shows the high level behavior of Ceph. > > Here's the IO path of a write_full() - how it goes from the client to all OSDs > and back. The OSDs talk to each other through their respective msg-worker > thread using sockets, and the threads of the same OSD talk to each other > through semaphores (int Wait(Mutex &mutex) in Cond.h). Colored arrows are > messages across nodes (over the network). Thick horizontal lines separate > nodes, thin horizontal lines separate threads on the same node. > https://docs.google.com/drawings/d/1fFb-aI8NQq7RESV2OKzrte2KvaBWAh6X9TL7woAGAmo/edit?usp=sharing > > I did the same exercise for librbd (build with -finstrument-functions). Here's > the IO path in librbd for writes against images with RBD journaling enabled > and disabled. The red parts only appear when journaling is enabled. There's > still a bit more differences for the final flush() part, but I've omitted them > for simplicity. If there's need/interest I can add those as well. The dotted > vertical line separate iterations of writes() in the client (rbd tool). > https://docs.google.com/drawings/d/1s9pRVygdPzBtQyYrvPueF8Cm6Xi1d4oy5gNMOjx2Wzo/edit?usp=sharing > > I think these are the kind of diagrams we'll be able to get more easily from > Zipkin/Blkin. Yes! sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
