On Sat, 2006-10-14 at 23:54 -0500, Mike McGrath wrote: > I've been working on some scripts for mining information out of our > mirror logs. In addition to the awstats scipts I'm keeping a total > archive of all requests in a special format. This will allow us to > mine the information as we need to. The current plan is to keep the > archives as is until it becomes unmanageable. At present about one > months worth of logs gets reduced to about 50 M, about 5M after > compression. > > In case anyone is wondering the logs are being kept on > fpserv:/srv/mirrorStats/stats.txt > > The format is: > > [08/Oct/2006:05:05:03 a05543977dc854b859da8e365de818b8 rawhide i386 > > its quick, and easy to run tools against. The second column is a md5 > encrypted version of the ip address. Its not perfect but at least its > not plain text. I'm sure if someone really wanted to convert these > back to IP's they could create a rainbow table or some such silliness > but this should pre-empt anyones fears about us stockpiling ip's or > any personal information for that matter. > > Eventually I'll write static frontend pages for this similar to our > awstats pages. For those keeping count there's been 17800 unique IP's > that have contacted the mirrors server looking for rawhide. maybe we could put the stats into rrd's -sv
