Angela Burrell wrote:
Squid users, I am running Ubuntu Edgy 6.10 for AMD64 which is Kernel 2.6.17. I have squid 2.6.STABLE9 that I installed from source. This is not a production server yet, so it was just kind of sitting there for the past few days, today when I turned on the screen I saw messages similar to the following: (numbers) Out of memory: Kill process XXX (apache2) score XX and children. (numbers) Out of Memory: Killed process XXX (apache2). (...) (numbers) Out of memory: Kill process XXX (squid) score XX and children. (numbers) Out of Memory: Killed process XXX (squid). I don't know too many linux commands, but I looked on google and found the free command. My computer has 1 GB of RAM and I only had 10 MB free, but I had about 75% of my swap space left. (I have about 3 GB of swap).
"Free" memory in Linux (and BSD, and Solaris, etc.) is a funny thing if you are used to the Windows world. The Gentoo Wiki has a nice explanation on this (http://gentoo-wiki.com/index.php?title=FAQ_Linux_Memory_Management&oldid=97182). Stop after the first paragraph as the rest is a bit irrelevant. The short answer is use "free -m" and look at the "+/- buffers/cache" line.
But by the time processes start to get killed, there is a real problem.
I did "/etc/init.d/squid stop" which replied [ok] but then when i look in "top" I see that I still only have 10 MB free memory, and most of the pid's on the list belong to squid. Why are 30,000 squid processes running after I type "squid stop".
That depends entirely on what the initialization script does. If it just runs /path/to/squid -k shutdown and checks the result of that call, any open connections will be given a certain amount of time (default is 30 seconds last time I checked). But 30,000 processes sounds like a lot (to put it mildly). Why are there 30,000 processes related to squid would be the question I would want answered.
According to ps aux, /usr/sbin/squid -D -sYC has 9486 entries (AFTER I typed squid stop.)
Yikes. If I'm not mistaken (which happens a lot more frequently than I would like) that means that squid was "started" over 9,000 times without any of them dying. I'm not even sure how that would be possible. Any time I try to start more than one instance of Squid the second one dies complaining that the port it wants to use can't be opened....
Assuming that it would be possible to start squid 9000 times that would explain the number of processes. Each parent Squid process usually kicks off the child process (that handles the requests) and an unlinkd process (which gets rid of expired cache objects). Optionally authentication processes and external_acl processes are also started.
On our production server we have 512 MB of RAM and squid 2.4.STABLE7 and this has never happened. What can I do? Note that during the last 5 days this computer was unplugged from the network so, it's cache cannot be full!
Monitor the cache.log. Make sure the pid file is successfully written. Check for any other funny business. Figure out how multiple thousand squid processes were started without interfering with each other and dying.
Chris
