On Thu, Mar 22, 2012 at 11:17, Mark Kampe <Mark.Kampe@xxxxxxxxxxxxx> wrote: > (a) I actually favor not simply mmaping the circular buffer, > but having a program that pulls the data out of memory > and writes it to disk (ala Varnish). In addition to doing > huge writes (greatly reducing the write overhead), it can > filter what it processes, so that we have extensive logging > for the last few seconds, and more manageable logs on disk > extending farther back in time (modulo log rotation). varnishlog communicates via a mmap'ed ringbuffer. These are not in conflict. > (b) The most interesting logs are probably the ones in coredumps > (that didn't make it out to disk) for which we want a > crawler/extractor anyway. It probably isn't very hard to > make the program that extracts logs from memory also be > able to pick the pockets of dead bodies (put a big self > identifying header on the front of each buffer). A single memory area for this makes it easier.. that's why I like the ringbuffer. > If we are logging a lot, buffer management has the potential > to become a bottle-neck ... so we need to be able to allocate > a record of the required size from the circular buffer > with atomic instructions (at least in non-wrap situations). > > But if records are allocated and then filled, we have to > consider how to handle the case where the filling is > delayed, and the reader catches up with an incomplete > log record (e.g. skip it, wait how long, ???). > > And while we hope this will never happen, we have to deal > with what happens when the writer catches up with the > reader, or worse, an incomplete log block ... where we might > have to determine whether or not the owner is deceased (making > it safe to break his record lock) ... or should we simply take > down the service at that point (on the assumption that something > has gone very wrong). The Disruptor design handles all these, is simple in the sense of "that's what I would have built", and looks good. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
