Thanks so much for all the advise and responses :-)
I decided to try Dansguardian.........
Currently I have a working model setup though it needs a bit of tuning
and tweaking but good news is that I am using the SquidGuard blacklists
so all is pretty much good!!
Have been testing; performance is phenomenal though sometimes when Squid
can't connect to a site properly in order to populate the cache etc...
the pages might need a bit of refreshing however, I consider those as
just teething problems.
So yeah.... NET <- NAT <- <Squid + c-icap + Clamd> <- Dansguardian <- PF
is how things look like now :-)
Regards,
Kaya
On 11/09/2013 10:37 PM, Marcus Kool wrote:
On Sat, Nov 09, 2013 at 11:16:12PM +0100, Loïc BLOT wrote:
Hello Kaya,
first, don't forget to look at sysctl kern.maxfiles values.
Also improve daemon FD values in login.conf for squid. Don't forget each
connection is a FD (1 connection for the client, 1 for the transaction
to remote site, somes for the caching).
Also to improve performances of squidguard, i stored all blacklists DB
to a memory fs (mfs) this improve massively squidguard performance
If the disk I/O is really the bottleneck, consider ufdbGuard.
ufdbGuard loads the URL database in memory and easily does
25,000 URL lookups/sec, much more than you will ever need.
Marcus
I have wrote an article to improve squid perfs on OpenBSD:
http://www.unix-experience.fr/2013/monter-un-proxy-cache-performant-avec-squid-et-openbsd/
--
Best regards,
Loïc BLOT,
UNIX systems, security and network engineer
http://www.unix-experience.fr
Le samedi 09 novembre 2013 à 19:39 +0000, Kaya Saman a écrit :
Just found this is Squid cache log:
2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of
redirector processes in your config file.
The cache size is 2GB.... though that shouldn't affect performance as
far as I understand.
On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
Hey,
Notes inside.
On 11/09/2013 05:58 PM, Kaya Saman wrote:
What can I do to improve performance with this?
Is this line too high: url_rewrite_children 500
YES!!
or simply have a misconfigured something?
I additionally have 'c-icap' running with squidclamav coupled to clamd
in case that is of importance - not using the squidGuard line in the
squidclamav.conf file!!!
Basically how can I get the IO usage down and get the system to work
again?
For how many users exactly?
Just a note that I am not in a favor of any OS by default but I would
feel better Using Linux.
- the logs don't indicate anything outside of 'starting squidGuard
process' many times.
The basic assumption of using 500 child process is that you have
atleast 100 CPUs.
SquidGuard was design for performance which is lots of urls per sec.
It can be tested just to clear the point out.
for example in a rate of 1500k requests per second you should not have
a need in more then 40-50 children.
In practice it works a bit different speed since there is a speed
limit on STDIN and STDOUT which slows down the speed of squid and
squidguard communication blocking the whole squid instance(in a way).
If you need basic url filtering you can use ICAP which has an option
to run as a standalone service outside of squid settings and machine.
I have written in the past a small ICAP service for the favor of
requests manipulation and filtering.
I have never finished it in a level I was happy with but the basic
code can be seen here:
https://github.com/elico/echelon
I know for a fact that ICAP interface adds concurrency by the "nature"
of it using TCP.
This is not the place to ask about concurrency in squidguard which can
allow the usage of square less processes(children) for more requests.
In order to find the right number of children start with 40 and see if
it fits you and then see what is the bottle neck in the whole setup.
Eliezer
