On Mon, 16 Nov 2009 16:40:16 +0100, "David B." <haazeloud@xxxxxxxxx> wrote: > Hi Squid users, > > We're using squid as a reverse proxy cache. Server (debian lenny) > running squid (from lenny stable / squid/3.0.STABLE8) seems to have some > stange behaviour. > For example, we've got lots of TIN_WAIT1 TCP Connexions and we can't > figure why. :( > > # netstat -na | wc -l > 20065 > > # netstat -na | grep 'FIN_WAIT1' | wc -l > 11255 > > Around 50% of TCP sessions are in "FIN_WAIT1", other 40% are "ETABLISHED". > > Theses connexions are from client of squid cache : > Exemple, T.T.T.T is my public interface. > x.y.z.w is a client IP v4 address. > > tcp 0 1 T.T.T.T:80 x.y.z.w:50530 FIN_WAIT1 > > Did someone notice this before ? I have not seen FIN_WAIT1 before, but often see FIN_WAIT like this when Squid is receiving a lot of connections. I think its due to squid using sockets for short times (non-persistent connections) and moving on. The system TCP timeouts are much longer. > > We thought this will not be squid related, but kernel related. I can > only see this on squid machines and not apache machines for example. :( > > We can also see errors in the kern.log like : > kernel: [1675388.847059] Out of socket memory > > This seems to be linked. I assumed i've got too much TCP sessions for > the kernel reserved memory. I agree. > > Squid is behind an LVS load balancer, do you think my TCP problem is > from there ? Because on the nex request, the same client could hit a > different squid machine. Maybe yes, maybe no. I assume the LVS works on TCP-link connections so one persistent connection is not broken up. Thats the only breakage that could make this issue worse. Normal LVS would be helping a bit by spreading the load off the problem Squid boxes. > > Perhaps my load is too high and i need to tune kernel via sysctl, but i > can't figure what to do. For now, i've tried several things and i can't > solved this issue. You may want to check: * persistent connections is turned on (squid.conf) * system socket timeouts * half-closed clients is off. (squid.conf) Hopefully someone with real finger-time at high performance systems will be able to point at specific tuning knobs which are helpful. > > Here's some squid stats : > HTTP/1.0 200 OK > Server: squid/3.0.STABLE8 > Mime-Version: 1.0 > Date: Mon, 16 Nov 2009 15:35:35 GMT > Content-Type: text/plain > Expires: Mon, 16 Nov 2009 15:35:35 GMT > Last-Modified: Mon, 16 Nov 2009 15:35:35 GMT > X-Cache: MISS from ww.xx.com > X-Cache-Lookup: MISS from ww.xx.com:80 > Connection: close > > Squid Object Cache: Version 3.0.STABLE8 > Start Time: Mon, 09 Nov 2009 09:03:56 GMT > Current Time: Mon, 16 Nov 2009 15:35:35 GMT > Connection information for squid: > Number of clients accessing cache: 254750 > Number of HTTP requests received: 337655902 > Number of ICP messages received: 0 > Number of ICP messages sent: 0 > Number of queued ICP replies: 0 > Number of HTCP messages received: 0 > Number of HTCP messages sent: 0 > Request failure ratio: 0.00 > Average HTTP requests per minute since start: 32244.7 avg 537 req/sec. Nice :) Are you able to provide CPU details sop we can add this to http://wiki.squid-cache.org/KnowledgeBase/Benchmarks ? Amos
