> -----Original Message----- > Subject: Re: Looking for web usage reporting solution > On Fri, Nov 13, 2009 at 11:31 AM, Aaron Spurlock > > I am looking for a web usage reporting solution that can run via > sniffing or from a mirror port on a switch. I envision this solution > would simply log each URL request it sees and allow reports to be > generated on web sites that internal users have gone to. I've searched > high and low, but cannot find a "ready-made" solution, so I'm looking > to put it together myself. > > > > Most people/posts suggest using squid/squidgard/dan's guardian, but > it appears to me that is only an inline solution, and I would prefer a > sniffing solution for safety (if machine crashes, it doesn't take down > Internet). In that sense, it would work a lot like websense, but > without the blocking, only reporting. > > > > From a high-level pseudo-code standpoint, it would simply sniff all > traffic, and when it sees a packet requesting a webpage, it parses it > and dumps these results into a database: > > > > -Date > > -Time > > -Source IP > > -Dest IP > > -URL requested > > -FQDN portion of web request - IE: if request was for > > http://www.microsoft.com/windows/server/2003, it records only > > www.microsoft.com here > > -domain portion of web request - only microsoft.com in above example > > > > Using this data, I can then produce reports for the client on who > went where when.... Personally, I thought this would be a great program > for open source, but I can't find anything like this already out > there!!! It seems like kind of a mix between Squid, NTOP and Snort... > > > > Thanks for any thoughts on this project! > > > > Sounds like you've got it pretty much worked out...not sure what the > question is? Sniffing for HTTP traffic should work fine, then you just > need to parse the traffic. I would probably just use iptables to sniff > for traffic on port 80 and 8080. Of course, you can't sniff SSL > communications, but there's really no solution to that unless you can > break the encryption (which you can't). I guess this is technically > still an inline solution, but short of actually buying a hardware > packet sniffer, I'm not sure that there is any "out of line" solution. LOL, just saw your post about the reply-all issue...I almost did that with this message! Anyway, back to the regularly scheduled program. Can I use IPtables to log those packets to a file that I could then parse? I've been thinking about doing it with tcpdump, but I'm not sure what kind of output I'd get!
