On Sat, Nov 09, 2013 at 11:16:12PM +0100, Loïc BLOT wrote: > Hello Kaya, > first, don't forget to look at sysctl kern.maxfiles values. > Also improve daemon FD values in login.conf for squid. Don't forget each > connection is a FD (1 connection for the client, 1 for the transaction > to remote site, somes for the caching). > > Also to improve performances of squidguard, i stored all blacklists DB > to a memory fs (mfs) this improve massively squidguard performance If the disk I/O is really the bottleneck, consider ufdbGuard. ufdbGuard loads the URL database in memory and easily does 25,000 URL lookups/sec, much more than you will ever need. Marcus > I have wrote an article to improve squid perfs on OpenBSD: > http://www.unix-experience.fr/2013/monter-un-proxy-cache-performant-avec-squid-et-openbsd/ > > > > -- > Best regards, > Loïc BLOT, > UNIX systems, security and network engineer > http://www.unix-experience.fr > > > > Le samedi 09 novembre 2013 à 19:39 +0000, Kaya Saman a écrit : > > Just found this is Squid cache log: > > > > 2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files > > 2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy. > > 2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued > > 2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of > > redirector processes in your config file. > > > > > > The cache size is 2GB.... though that shouldn't affect performance as > > far as I understand. > > > > On 11/09/2013 05:23 PM, Eliezer Croitoru wrote: > > > Hey, > > > > > > Notes inside. > > > > > > On 11/09/2013 05:58 PM, Kaya Saman wrote: > > >> > > >> What can I do to improve performance with this? > > >> > > >> > > >> Is this line too high: url_rewrite_children 500 > > > YES!! > > > > > >> or simply have a misconfigured something? > > > > > > > > >> I additionally have 'c-icap' running with squidclamav coupled to clamd > > >> in case that is of importance - not using the squidGuard line in the > > >> squidclamav.conf file!!! > > >> > > >> Basically how can I get the IO usage down and get the system to work > > >> again? > > > For how many users exactly? > > > Just a note that I am not in a favor of any OS by default but I would > > > feel better Using Linux. > > > > > >> > > >> - the logs don't indicate anything outside of 'starting squidGuard > > >> process' many times. > > > The basic assumption of using 500 child process is that you have > > > atleast 100 CPUs. > > > SquidGuard was design for performance which is lots of urls per sec. > > > It can be tested just to clear the point out. > > > for example in a rate of 1500k requests per second you should not have > > > a need in more then 40-50 children. > > > In practice it works a bit different speed since there is a speed > > > limit on STDIN and STDOUT which slows down the speed of squid and > > > squidguard communication blocking the whole squid instance(in a way). > > > > > > If you need basic url filtering you can use ICAP which has an option > > > to run as a standalone service outside of squid settings and machine. > > > > > > I have written in the past a small ICAP service for the favor of > > > requests manipulation and filtering. > > > I have never finished it in a level I was happy with but the basic > > > code can be seen here: > > > https://github.com/elico/echelon > > > > > > I know for a fact that ICAP interface adds concurrency by the "nature" > > > of it using TCP. > > > > > > This is not the place to ask about concurrency in squidguard which can > > > allow the usage of square less processes(children) for more requests. > > > > > > In order to find the right number of children start with 40 and see if > > > it fits you and then see what is the bottle neck in the whole setup. > > > > > > Eliezer > >
