On Tue, 24 Jul 2007, Adrian Chadd wrote:
On Mon, Jul 23, 2007, gonzales@xxxxxxxxxxxxxx wrote:
Squid 2.6.stable14
I've got a small dillema. I've written an external Perl helper to return
"OK" or "ERR" dependant upon some regular expressions and/or domains
stored in an external Postgresql database.
Pretty simple.
What I've noticed is that for each URL/FQDN that is requested, squid
passes 'every' URL embedded in the webpage, one by one, to the external
helper(s that are running, until I add the asynchronous piece that allows
just one instance of the helper to run, which will then fork children
processes as needed). My external helper, makes a call to a postgresql db
and checks each domain/url against a single table of 50K entries, this
KILLS the performance and end user experience.
Yup, run multiple helpers.
Multiple or single with ansychronous threads, being better? I would think
that running with simply a rigid number of helpers would constrain
performance, in the case of the N+1 request, where the first N helper
instances are 'busy' working. Would think that single helper that can
fork and deal with IPC would be better suited. Thoughts on this?
>
Does anyone have any suggestions as to 'how' to make this work well? Or
Henrik, do you have any suggestions as to where I might start looking in
the Squid code, as to how I can modify 'how' URL's are passed to the
external helper?
See why the Postgresql lookups are taking so long. me, I'm doing
this stuff in my spare time by building DBM type files (currently playing
with TDB) and I'm able to do rather large hash table lookups rather quickly.
SQL isn't necessarily the right tool for this.
I don't think it's the lookup, explain:
bandaboo1=> explain select * from black_list_fqdn ;
QUERY PLAN
------------------------------------------------------------------------
Seq Scan on black_list_fqdn (cost=0.00..5648.59 rows=311559 width=32)
(1 row)
I think it's how many/how frequent Squid has to make entries into the
helper, is there a way for Squid to go the URL requested by the user, and
send 'every' URL from the page to the external helper at once? Rather
than one at a time, return value, enter again, ... ad infinitum.
Adrian,
thanks for initial feedback!
> >
Adrian
--
Louis Gonzales
louis.gonzales@xxxxxxxxxxxxxx
http://www.linuxlouis.net
