Solution: preprocessing logs Great ideas, I'll look into some of these, thanks very much. > We have a cluster of 6 machines, some running Apache, some running MySQL. > We use shared logging successfully along with stats and post-processing > scripts. We also use plain-ol' logrotate with our shared logs. > > We use network-enabled syslog to capture logging on every node to a single, > master logging node (with fail-over, of course!) > > For Apache, we use custom ErrorLog, CustomLog, and RewriteLog directives > per vhost to pipe output to a custom script which greps a few undesirable > statements out prior to logging. > > Apache is sent to the local1 facility on the target syslog > machine that holds all of our logs, where it's configured > with something like: > > /etc/syslog.conf: > # Cluster Apache Logging > local1.err /var/log/shared-apache-err.log > local1.notice /var/log/shared-apache-access.log > local1.debug /var/log/shared-apache-rewrite.log > > > And, for example, all Apache nodes use the same config akin to: > > /path/to/http-vhost.conf: > <snip> > ErrorLog "|/path/to/logger.pl err some_string_ID" > CustomLog "|/path/to/logger.pl notice some_string_ID" > RewriteLog "|/path/to/logger.pl debug some_string_ID" > </snip> > > where logger.pl continually reads input, runs some filters > to determine if it should indeed log the particular message, > and then calls Sys::Syslog's "syslog()" function, and > "some_string_ID" is a tag to identify each message in > the shared log files. > > You could really use any line-by-line filtering program > here, but be aware that Apache executes the first argument > after the pipe symbol directly - it doesn't run a shell or > anything, so you don't have any expansion, piping of other > commands, etc. > > You can also use /usr/bin/logger (see "man logger") to > send output to various facilities (localN) and informational > levels (err, notice, debug, etc.). This does the same > thing as "logger.pl" above, but doesn't provide any > filtering. > > Also, we've seen syslog drop some messages under > heavy load (hence why we filter some Apache logging > prior to syslogging it). I don't know the exact > cause - maybe someone else can shed light on that for me! > > > Hope this helps - it's what we do and it seems to work > well enough for what we need. > > Regards, > -Brenton Rothchild -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
