David B. wrote:
Hi Amos,
Amos Jeffries a écrit :
On Mon, 16 Nov 2009 16:40:16 +0100, "David B." <haazeloud@xxxxxxxxx>
wrote:
[Snip]
I have not seen FIN_WAIT1 before, but often see FIN_WAIT like this when
Squid is receiving a lot of connections.
I think its due to squid using sockets for short times (non-persistent
connections) and moving on. The system TCP timeouts are much longer.
Maybe, it's quite strange.
[Snip]
Maybe yes, maybe no.
I assume the LVS works on TCP-link connections so one persistent
connection is not broken up. Thats the only breakage that could make this
issue worse.
Normal LVS would be helping a bit by spreading the load off the problem
Squid boxes.
Sorry i haven't give you all details.
In fact, i've got an LVS in front, acting as a TCP/IP load balancer, no
persistent session there, it's a round robin.
Then, TCP request is transmitted to a squid box. This box will answer to
the client directly (DSR or direct routing mode).
I'm just wondering if activate "session" on the LVS to force someone to
join the same squid boxe will help.
Perhaps my load is too high and i need to tune kernel via sysctl, but i
can't figure what to do. For now, i've tried several things and i can't
solved this issue.
You may want to check:
* persistent connections is turned on (squid.conf)
On by défault i think, but i will force this to on to test.
* system socket timeouts
System socket or squid related like persistent_request_timeout ?
System socket. I'm guessing the FIN_WAIT* is created once Squid sends
its close signal to the system.
* half-closed clients is off. (squid.conf)
We've tried on and off on several boxes, no change for now.
Hopefully someone with real finger-time at high performance systems will
be able to point at specific tuning knobs which are helpful.
I hope too.
I can give the squid conf if needed. :)
Here's some squid stats :
HTTP/1.0 200 OK
Server: squid/3.0.STABLE8
Mime-Version: 1.0
Date: Mon, 16 Nov 2009 15:35:35 GMT
Content-Type: text/plain
Expires: Mon, 16 Nov 2009 15:35:35 GMT
Last-Modified: Mon, 16 Nov 2009 15:35:35 GMT
X-Cache: MISS from ww.xx.com
X-Cache-Lookup: MISS from ww.xx.com:80
Connection: close
Squid Object Cache: Version 3.0.STABLE8
Start Time: Mon, 09 Nov 2009 09:03:56 GMT
Current Time: Mon, 16 Nov 2009 15:35:35 GMT
Connection information for squid:
Number of clients accessing cache: 254750
Number of HTTP requests received: 337655902
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 32244.7
avg 537 req/sec. Nice :)
Are you able to provide CPU details sop we can add this to
http://wiki.squid-cache.org/KnowledgeBase/Benchmarks ?
Squid box is a VM under Xen.
CPU : 2 vcpus (xen), like 2 cores of an intel Xéon E5430 2.66 Ghz
Memory : 10 Go
Disks : VM storage on a hardware RAID 5 controller.
FS for cache : XFS with noatime,nodev,nosuid,noexec
Under peak, CPU is low : (max is 200%, 100% per core)
idle : 160%
iowait : 30%
system & user : 10%
I hope this will help.
David.
Wonderful. Thanks.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
Current Beta Squid 3.1.0.14
