On Sat, Sep 30, 2006 at 04:19:13PM +0200, Christoph Haas wrote: > On Saturday 30 September 2006 05:11, Chuck Kollars wrote: > > Our experience with web filtering is the differences > > in tools are _completely_ swamped by the quality and > > depth of the blacklists. (The reverse of course is > > also true: lack of good blacklists will doom_any_ > > filtering tool.) > > > > We currently have over 500,000 (!) sites listed in > > just the porn section of our blacklist. With quality > > lists like these, any old tool will do a decent job. > > And large portions of those half million sites are probably already > something different but porn sites or the domains were given up. I > wouldn't judge the quality completely by the quantity. > > > Lots of folks need to get such lists reasonably and > > regularly (quarterly?). > > Daily even. > > > Useful lists are far far too > > large to be maintained by local staff. Probably what's > > needed is a mechanism whereby everybody nationwide > > contributes, some central site consolidates and > > sanitizes, and then publishes the lists. > > I'd welcome such an effort. Some companies invest a lot of effort into URL > categorisation - not just regarding porn sites. But they have several > employees working full-time on that and run a kind of editor's office. > For a free/open-source project you would need a lot of people and some > mechanism (e.g. a web spider) that searches for further sites. And doing > that job is boring. So compared to other free/open-source projects there > is much less motivation to contribute constantly. > > > This would be a huge effort. It's not easily possible > > even with lots of clever scripts and plenty of compute > > power. We've already seen more than a handful of > > "volunteers" swallowed up by similar efforts. > > I believe that the only blacklist that survived over the ages was > http://urlblacklist.com/ - just that they are non-free now. I may be > mistaken about its history though. > > There already exist DNS-based blacklists that are very effective for mail > spam detection. Perhaps a DNS-based register where you can look up if a > certain domain belongs to a certain category might help. Large > installations like ISPs could mirror the DNS zone and private people could > just use them. Perhaps even the Squid developers could support such a > blacklist. > > So IMHO we lack both a source (volunteers, spider, web based contribution > system) and a good way to use it. Huge static ACLs don't work well with > Squid. > > Since I had to tell our managers at work on how well URL filtering works > (we use a commercial solution) I pulled some numbers. And around 3,000 > domains are registered at the DeNIC (german domain registry) alone every > day. Now try that with other registries and get a rough number on many > domains need to get categorized every day. That's the reason why it's so > hard to create reasonable blacklists. (And also the cause for my rants > where people expect decent filtering by just using the current publicly > available blacklists). > > You didn't tell much about your intentions though. :) > > Kindly > Christoph As was already stated above...sites come and go daily. It would very difficult to keep any current list available. Maybe updating on a daily basis and even then you would be behind. Perhaps in addition to squid, you could place other filtering software to check the web page language. I use Squid + Dansguardian for my children. You could look for something like it. I only block know URLs that are less likely to change over night...example, playboy. It is not a perfect solution, but it helps keep my URL list down. Wish you well. -- Alex FreeBSD 6.0-RELEASE i386 GENERIC
