Hi Squid users,
We're using squid as a reverse proxy cache. Server (debian lenny)
running squid (from lenny stable / squid/3.0.STABLE8) seems to have some
stange behaviour.
For example, we've got lots of TIN_WAIT1 TCP Connexions and we can't
figure why. :(
# netstat -na | wc -l
20065
# netstat -na | grep 'FIN_WAIT1' | wc -l
11255
Around 50% of TCP sessions are in "FIN_WAIT1", other 40% are "ETABLISHED".
Theses connexions are from client of squid cache :
Exemple, T.T.T.T is my public interface.
x.y.z.w is a client IP v4 address.
tcp 0 1 T.T.T.T:80 x.y.z.w:50530 FIN_WAIT1
Did someone notice this before ?
We thought this will not be squid related, but kernel related. I can
only see this on squid machines and not apache machines for example. :(
We can also see errors in the kern.log like :
kernel: [1675388.847059] Out of socket memory
This seems to be linked. I assumed i've got too much TCP sessions for
the kernel reserved memory.
Squid is behind an LVS load balancer, do you think my TCP problem is
from there ? Because on the nex request, the same client could hit a
different squid machine.
Perhaps my load is too high and i need to tune kernel via sysctl, but i
can't figure what to do. For now, i've tried several things and i can't
solved this issue.
Here's some squid stats :
HTTP/1.0 200 OK
Server: squid/3.0.STABLE8
Mime-Version: 1.0
Date: Mon, 16 Nov 2009 15:35:35 GMT
Content-Type: text/plain
Expires: Mon, 16 Nov 2009 15:35:35 GMT
Last-Modified: Mon, 16 Nov 2009 15:35:35 GMT
X-Cache: MISS from ww.xx.com
X-Cache-Lookup: MISS from ww.xx.com:80
Connection: close
Squid Object Cache: Version 3.0.STABLE8
Start Time: Mon, 09 Nov 2009 09:03:56 GMT
Current Time: Mon, 16 Nov 2009 15:35:35 GMT
Connection information for squid:
Number of clients accessing cache: 254750
Number of HTTP requests received: 337655902
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 32244.7
Average ICP messages per minute since start: 0.0
Select loop called: 1720466448 times, 0.365 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 97.6%, 60min: 97.9%
Hits as % of bytes sent: 5min: 96.0%, 60min: 96.5%
Memory hits as % of hit requests: 5min: 73.8%, 60min: 74.8%
Disk hits as % of hit requests: 5min: 4.9%, 60min: 4.9%
Storage Swap size: 60397908 KB
Storage Swap capacity: 90.0% used, 10.0% free
Storage Mem size: 4194284 KB
Storage Mem capacity: 100.0% used, 0.0% free
Mean Object Size: 67.08 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.00000 0.00000
Cache Misses: 0.00091 0.00091
Cache Hits: 0.00000 0.00000
Near Hits: 0.00091 0.00091
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.00000 0.00000
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 628299.345 seconds
CPU Time: 68051.533 seconds
CPU Usage: 10.83%
CPU Usage, 5 minute avg: 9.56%
CPU Usage, 60 minute avg: 8.77%
Process Data Segment Size via sbrk(): 1027896 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 2
Memory usage for squid via mallinfo():
Total space in arena: 862404 KB
Ordinary blocks: 794528 KB 53203 blks
Small blocks: 0 KB 0 blks
Holding blocks: 379248 KB 2048 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 67875 KB
Total in use: 1173776 KB 95%
Total free: 67875 KB 5%
Total size: 1241652 KB
Memory accounted for:
Total accounted: 895030 KB 72%
memPool accounted: 895030 KB 72%
memPool unaccounted: 346621 KB 28%
memPoolAlloc calls: 53284818867
memPoolFree calls: 53271070253
File descriptor usage for squid:
Maximum number of file descriptors: 65535
Largest file desc currently in use: 52590
Number of file desc currently in use: 50471
Files queued for open: 1
Available number of file descriptors: 15063
Reserved number of file descriptors: 100
Store Disk files open: 42769
Internal Data Structures:
907161 StoreEntries
243509 StoreEntries with MemObjects
243363 Hot Object Cache Items
900335 on-disk objects
Thank you.
David.
