Re: Reliable combined log sorting

[Eike Frost <ei@xxxxxxxx>]

Hi Jason,

On Sun, May 9, 2010 at 5:41 AM, Jason Nunnelley <jason@xxxxxxxxxx> wrote:

In our configuration of Apache we push all logs to a single file via syslog-ng. I need to create some basic log analysis (like AWstats style) for a single domain out of several domains that post to that file. Any suggestions on a simple solution?

What LogFormat are you using, precisely ? And what, precisely, is the problem you are having (you mention sorting in the subject, but nothing related to it in the body).

If all you require is the ability to filter for a single vhost, have a look at http://httpd.apache.org/docs/2.2/mod/mod_log_config.html, make sure your LogFormat-string contains "%v" somewhere, and filter by the respective field in the syslog-ng file -- which can trivially be done with awk or similar. If your line looks something like this :

yourvhost.example.com 66.249.xx.xx - - [09/May/2010:15:25:56 +0000] "GET /xxx HTTP/1.1" 200 8284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 261 8619

due to a LogFormat of  "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combined_2 (or something similar), then 

awk '$1 ~ /yourvhost\.example\.com/ { print }' < logfilename

will filter out just those lines where the first field ($1) matches your VirtualHost ServerName of yourvhost.example.com -- which you are then free to pipe to whatever log analysis method you want. If you want this to also filter out the syslog-related timestamps, awk can do that as well; http://www.gnu.org/manual/gawk/gawk.html should provide sufficient detail on how manipulate the print-statement.

If your problem is that the log entries are not in strictly chronological order (which most decent log analysis software can deal with as long as the delta is not too large), the problem becomes different beast altogether -- one which you'd likely want to employ a more sophisticated perl or python script since you require more state -- code for which I don't have handy at the moment ;-)

If this does not answer your question, please provide some more example log and code.

--Eike