I've been working on some scripts for mining information out of our
mirror logs. In addition to the awstats scipts I'm keeping a total
archive of all requests in a special format. This will allow us to
mine the information as we need to. The current plan is to keep the
archives as is until it becomes unmanageable. At present about one
months worth of logs gets reduced to about 50 M, about 5M after
compression.
In case anyone is wondering the logs are being kept on
fpserv:/srv/mirrorStats/stats.txt
The format is:
[08/Oct/2006:05:05:03 a05543977dc854b859da8e365de818b8 rawhide i386
its quick, and easy to run tools against. The second column is a md5
encrypted version of the ip address. Its not perfect but at least its
not plain text. I'm sure if someone really wanted to convert these
back to IP's they could create a rainbow table or some such silliness
but this should pre-empt anyones fears about us stockpiling ip's or
any personal information for that matter.
Eventually I'll write static frontend pages for this similar to our
awstats pages. For those keeping count there's been 17800 unique IP's
that have contacted the mirrors server looking for rawhide.
-Mike
