Ralf Hildebrandt wrote:
* Amos Jeffries <squid3@xxxxxxxxxxxxx>:
Ralf Hildebrandt wrote:
3.0.STABLE25 is showing the following behaviour during normal operation:
Hi Ralf,
Thank you for all this, but I'm wondering why you are putting so
much work into 3.0?
3.1 sucks even more? See my other bug reports! That stuff is crashing
all over the place. Need to get some stability here :)
:(
I ask because the major performance gains are aimed at 3.2. It could
do with this type of analysis as part of the polish up.
I COULD run 3.2 if you like.
:)
Okay, add to that a sudden extreme loss of known clients. And a
sudden 'instant' drop in memory usage before the growth.
This looks to me like the usual culprit:
A Squid crash followed by dirty rebuild of a large caches' index.
Could be
The behaviour in such a situation is complete non-response on the
ports for a short period (extreme service times for existing clients,
they simply get no further traffic and time out).
Yes, but that should only be a problem for the clients.
Followed by a period of heavy reads as the entire cache_dir get
scanned file-by-file for meta data to build the index. Some of which
will fail as the un-closed files from previous instance are found.
Accompanied by heavy writes as the swap.state journal gets rebuilt
from each of those meta-data reads.
Under heavy client load this extra disk IO can lead to delays
processing other actions and slower new client service times.
OK, but for such a long time?
Yes. Depending on the cache size. Some people have reported it taking
IIRC a dozen minutes or more for GB+ caches.
If I'm reading those graph scales right your time scale was ~20 minutes
before the server maxed out to overload?
Potentially a huge backlog of buffered in-transit data waiting to be
stored in the cache. Which can't be written to until the index is
loaded properly.
This latter can be alleviated by a sufficiently large in-memory
cache, though older versions did not permit that space to be used
until after the rebuild either.
proxy 10426 88.5 17.3 380636 357040 ? R 10:42 110:54 /usr/sbin/squid3 -NsYC
% strace -c -p 10426
Over how long a time was the strace taken? just that 1.6 seconds or
something longer?
That trace is from about 15 seconds.
Ah so 99% unknown operations inside Squid.
The 24K reads + 24K writes equates to (@4KB pages) roughly 6.25Mbps in
IO each way. Assuming that the network traffic takes up a portion up to
~1/2 thats still a lot of disk IO to sustain.
Does your cache.log show any indication of a crash leading up to all this?
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.6
Beta testers wanted for 3.2.0.1
