The poor's man method: pipe the logs in a script, detect patterns of what annoys you, send an snmp message (or a mail).
I used a similar technique to generate syslog events that were captured by TNG.
Below an excerpt from what I documented for my ops team (does not render well in pure text - tell me if you want it in other format):
Appendix G: setup apache for TNG Syslog integration
The apache config must be modified to pipe the error log into a script. This script takes as input the name of the instance to add in the syslog and the location of a configuration file containing the filters to convert apache events into TNG events.
The output of the script must be redirected to the error log so normal logging still occurs.
The entry
ErrorLog /var/log/apache2-snoopy1/error.log
becomes
ErrorLog "|/usr/bin/apachelogs2tng snoopy1 /etc/apache2-snoopy1/tngevents.conf >> /var/log/apache2-snoopy1/error.log"
The configuration file must state the address of the syslog collector and a hash table with the patterns to detect and TNG messages.
The config file (name it as you want) must look like what follows:
$syslog = 'name-of-your-syslog-server';
%filters =
(
'ap_proxy_connect_backend disabling worker' => 'SYS-Unknown-APACHE 34-001',
'ajp_ilink_receive failed' => 'SYS-Unknown-APACHE 34-001'
);
0;
In this example, a backend unavailable error will be detected and reported into the log files with the associated code. The instance name and original message is added.
The messages that will appear in the syslog is show below:
Apr 16 13:17:20 snoopy snoopy1: SYS-Unknown-APACHE 34-001 (snoopy1) ap_proxy_connect_backend disabling worker for (127.0.0.1)
The source code of the script is given below. Care as been taken to filter user supplied data and log entries to avoid any injection through this thing.
#!/usr/bin/perl -wT
# -------------------------------------------------------
# Filter apache error log to insert TNG-formated messages
# into the syslog so TNG agent can catch them
#
# LDE for Mobistar IAG 2009
# -------------------------------------------------------
use strict;
use Sys::Syslog qw(:DEFAULT setlogsock);
our %filters;
our $syslog;
$|=1;
my $instance = $ARGV[0];
my $filterfile = $ARGV[1];
my $msg;
if( $filterfile =~ /^([a-zA-Z0-9_\-\/\.]+)$/ ) {
$filterfile = "$1";
}
else {
print "Unsafe config file";
exit 1;
}
do $filterfile;
setlogsock( "udp" );
$Sys::Syslog::host = $syslog;
openlog( sprintf( "%s", $instance ), '', 'LOCAL6' );
while( <STDIN> ) {
# output to STDOUT so apache logs it to logfile
$msg = sprintf( "%s", $_ );
print $msg;
# extract message from log line, pass it through filter and syslog if match
$msg =~ s/^.+\] //;
foreach my $key ( keys %filters ) {
if( m/$key/ ) {
$msg = sprintf( "%s (%s) %s", $filters{$key}, $instance, $msg );
syslog( 'notice', $msg );
last;
}
}
}
closelog;
exit 0;
The script is started for each ErrorLog directive in Apache configuration files and stays alive during the entire Apache process life. So the overhead is not spent for each log entry.
The target system must have the TNG syslog agent installed so it can capture and report the events.
The reference file contains the patterns and TNG events plus a category for the events. A reverse-proxy has additional events to trap compared to a static pages web server. A web server where Siteminder is installed should use the Siteminder filter/entries too.
-----Original Message-----
From: Miguel González Castaños [mailto:miguel_3_gonzalez@xxxxxxxx]
Sent: Thursday 23 February 2012 22:25
To: users@xxxxxxxxxxxxxxxx
Subject: Re: checking logs
On 23/02/2012 22:14, J.Lance Wilkinson wrote:
> Miguel González Castaños wrote:
>> Hi,
>>
>> Is there any tool that performs automatic checks against Apache (or
>> Tomcat) log files? I want to be able to monitor when something is
>> going wrong that needs attention from me instead of reading all logs
>> from Apache.
>>
>> I'm thinking of a crontab job or something similar
>>
>> Regards,
>>
>> Miguel
>
> What about NAGIOS, it can watch just about anything dynamically.
>
Well, I was referring more a script that watches the error and access logs and report in any suspicious entry
Regards,
Miguel
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
*****DISCLAIMER*****
This electronic transmission (and any attached document) is intended
exclusively for the person or entity to whom it is addressed and may
contain confidential and/or privileged material.
Any disclosure, copying, distribution or other action based upon
the information by persons or entities other than the intended recipient
is prohibited. If you receive this message in error, please contact the
sender and delete the material from any and all computers.
Mobistar does not warrant a proper and complete transmission of this
information, nor does it accept liability for any delays.
*****END OF DISCLAIMER*****
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|