Just found this is Squid cache log:
2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of
redirector processes in your config file.
The cache size is 2GB.... though that shouldn't affect performance as
far as I understand.
On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
Hey,
Notes inside.
On 11/09/2013 05:58 PM, Kaya Saman wrote:
What can I do to improve performance with this?
Is this line too high: url_rewrite_children 500
YES!!
or simply have a misconfigured something?
I additionally have 'c-icap' running with squidclamav coupled to clamd
in case that is of importance - not using the squidGuard line in the
squidclamav.conf file!!!
Basically how can I get the IO usage down and get the system to work
again?
For how many users exactly?
Just a note that I am not in a favor of any OS by default but I would
feel better Using Linux.
- the logs don't indicate anything outside of 'starting squidGuard
process' many times.
The basic assumption of using 500 child process is that you have
atleast 100 CPUs.
SquidGuard was design for performance which is lots of urls per sec.
It can be tested just to clear the point out.
for example in a rate of 1500k requests per second you should not have
a need in more then 40-50 children.
In practice it works a bit different speed since there is a speed
limit on STDIN and STDOUT which slows down the speed of squid and
squidguard communication blocking the whole squid instance(in a way).
If you need basic url filtering you can use ICAP which has an option
to run as a standalone service outside of squid settings and machine.
I have written in the past a small ICAP service for the favor of
requests manipulation and filtering.
I have never finished it in a level I was happy with but the basic
code can be seen here:
https://github.com/elico/echelon
I know for a fact that ICAP interface adds concurrency by the "nature"
of it using TCP.
This is not the place to ask about concurrency in squidguard which can
allow the usage of square less processes(children) for more requests.
In order to find the right number of children start with 40 and see if
it fits you and then see what is the bottle neck in the whole setup.
Eliezer
