On 01/05/12 20:09, Colin McCabe wrote:
Getting the system time is a surprisingly expensive operation, and this poses a problem for logging system designers. You can use the rdtsc CPU instruction, but unfortunately on some architectures CPU frequency scaling makes it very inaccurate. Even when it is accurate, it's not synchronized between multiple CPUs. Another option is to without time for most messages and just have a periodic timestamp that gets injected every so often-- perhaps every second, for example.
I agree it needs to be cheap ... but my experience with debugging problems in this sort of system suggests that we need the finest grained timestamps we can get (on every single message). Even though the clocks on different nodes are not that closely synchronized, computing the relative offsets from initial transactions isn't hard ... and then it becomes possible to construct a total ordering of events with accurate timings.
Pantheios and log4cpp are two potential candidates. I don't know that much about either, unfortunately.
Good suggestions. I am also looking at varnish (suggested by Wido den Hollander) which does logging in a shared memory segment from which external processes can save it (or not). What was (for me) a new idea here is the clean decoupling of in-memory event capture from on-disk persistence and log rotation. After I thought about it for a few minutes, I concluded it had many nice consequences.
Honestly, I feel like logging is something that you will end up customizing pretty extensively to suit the needs of your application. But perhaps it's worth checking out what these libraries provide-- especially in terms of efficiency.
I agree that the data captured is going to be something we hone based on experience. I am (for now) most concerned with the mechanism ... because I don't want to start making big investments in instrumentation until we have a good mechanism, around which we can fix the APIs that instrumentation will use. I'll try to review the suggested candidates and describe the mechanisms and advantages of each in another two weeks. Thank you very much for the feedback. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
