Search squid archive

Re: Squid and Squidguard using high disk IO

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Kaya,
first, don't forget to look at sysctl kern.maxfiles values. 
Also improve daemon FD values in login.conf for squid. Don't forget each
connection is a FD (1 connection for the client, 1 for the transaction
to remote site, somes for the caching).

Also to improve performances of squidguard, i stored all blacklists DB
to a memory fs (mfs) this improve massively squidguard performance
I have wrote an article to improve squid perfs on OpenBSD:
http://www.unix-experience.fr/2013/monter-un-proxy-cache-performant-avec-squid-et-openbsd/



-- 
Best regards,
Loïc BLOT, 
UNIX systems, security and network engineer
http://www.unix-experience.fr



Le samedi 09 novembre 2013 à 19:39 +0000, Kaya Saman a écrit :
> Just found this is Squid cache log:
> 
> 2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
> 2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
> 2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
> 2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of 
> redirector processes in your config file.
> 
> 
> The cache size is 2GB.... though that shouldn't affect performance as 
> far as I understand.
> 
> On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
> > Hey,
> >
> > Notes inside.
> >
> > On 11/09/2013 05:58 PM, Kaya Saman wrote:
> >>
> >> What can I do to improve performance with this?
> >>
> >>
> >> Is this line too high: url_rewrite_children  500
> > YES!!
> >
> >> or simply have a misconfigured something?
> >
> >
> >> I additionally have 'c-icap' running with squidclamav coupled to clamd
> >> in case that is of importance - not using the squidGuard line in the
> >> squidclamav.conf file!!!
> >>
> >> Basically how can I get the IO usage down and get the system to work 
> >> again?
> > For how many users exactly?
> > Just a note that I am not in a favor of any OS by default but I would 
> > feel better Using Linux.
> >
> >>
> >> - the logs don't indicate anything outside of 'starting squidGuard
> >> process' many times.
> > The basic assumption of using 500 child process is that you have 
> > atleast 100 CPUs.
> > SquidGuard was design for performance which is lots of urls per sec.
> > It can be tested just to clear the point out.
> > for example in a rate of 1500k requests per second you should not have 
> > a need in more then 40-50 children.
> > In practice it works a bit different speed since there is a speed 
> > limit on STDIN and STDOUT which slows down the speed of squid and 
> > squidguard communication blocking the whole squid instance(in a way).
> >
> > If you need basic url filtering you can use ICAP which has an option 
> > to run as a standalone service outside of squid settings and machine.
> >
> > I have written in the past a small ICAP service for the favor of 
> > requests manipulation and filtering.
> > I have never finished it in a level I was happy with but the basic 
> > code can be seen here:
> > https://github.com/elico/echelon
> >
> > I know for a fact that ICAP interface adds concurrency by the "nature" 
> > of it using TCP.
> >
> > This is not the place to ask about concurrency in squidguard which can 
> > allow the usage of square less processes(children) for more requests.
> >
> > In order to find the right number of children start with 40 and see if 
> > it fits you and then see what is the bottle neck in the whole setup.
> >
> > Eliezer
> 

Attachment: signature.asc
Description: This is a digitally signed message part


[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux