Squid 2.7.STABLE1-20080528 on Debian Linux 2.6.19.7
running on quad dual-core 2.6mhz Opterons with 32 gig RAM; 8x140GB disk
partitions
using WCCP L2 redirects transparently from a Cisco 4948 GigE switch
Server has one GigE NIC for the incoming redirects and two GigE NICs for
outbound http requests.
Using IPTables to port forward HTTP to Squid; no ICP, auth, etc.;
strictly a web cache using heap/LFUDA replacement
and 16GB memory allocated with mem pools on, no limit.
Used in an ISP environment, accommodating approx. 8k predominately cable
modem customers during peak.
Issue we're experiencing is some web pages taking in excess of 20
seconds to load, marked latency for customers
running web-based speed tests, etc.
Cache.log and Access.log aren't indicating any errors or timeouts;
system operates 96 DNS instances and 32k file descriptors
(neither has gotten maxed yet).
General Runtime Info from Cachemgr taken during pre-peak usage:
Start Time: Tue, 23 Sep 2008 18:07:37 GMT
Current Time: Tue, 23 Sep 2008 21:00:49 GMT
Connection information for squid:
Number of clients accessing cache: 3382
Number of HTTP requests received: 2331742
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 13463.4
Average ICP messages per minute since start: 0.0
Select loop called: 11255153 times, 0.923 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 42.6%, 60min: 40.0%
Byte Hit Ratios: 5min: 21.2%, 60min: 18.6%
Request Memory Hit Ratios: 5min: 18.3%, 60min: 17.2%
Request Disk Hit Ratios: 5min: 33.6%, 60min: 33.3%
Storage Swap size: 952545580 KB
Storage Mem size: 8237648 KB
Mean Object Size: 40.43 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.19742 0.12106
Cache Misses: 0.27332 0.17711
Cache Hits: 0.08265 0.03622
Near Hits: 0.27332 0.16775
Not-Modified Replies: 0.02317 0.00865
DNS Lookups: 0.09535 0.04854
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 10391.501 seconds
CPU Time: 4708.150 seconds
CPU Usage: 45.31%
CPU Usage, 5 minute avg: 33.29%
CPU Usage, 60 minute avg: 33.36%
Process Data Segment Size via sbrk(): 1041332 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 4
Memory usage for squid via mallinfo():
Total space in arena: 373684 KB
Ordinary blocks: 372642 KB 809 blks
Small blocks: 0 KB 0 blks
Holding blocks: 216088 KB 21 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 1041 KB
Total in use: 588730 KB 100%
Total free: 1041 KB 0%
Total size: 589772 KB
Memory accounted for:
Total accounted: 11355185 KB
memPoolAlloc calls: 439418241
memPoolFree calls: 378603777
File descriptor usage for squid:
Maximum number of file descriptors: 32000
Largest file desc currently in use: 9171
Number of file desc currently in use: 8112
Files queued for open: 2
Available number of file descriptors: 23886
Reserved number of file descriptors: 100
Store Disk files open: 175
IO loop method: epoll
Internal Data Structures:
23570637 StoreEntries
532260 StoreEntries with MemObjects
531496 Hot Object Cache Items
23561001 on-disk objects
Generated Tue, 23 Sep 2008 21:00:47 GMT, by
cachemgr.cgi/2.7.STABLE1-20080528@xxxxxxxxxxxxxxxxxx
TCPDUMP shows packets traversing all interfaces as expected; bandwidth
to both upstream providers isn't being maxed
and when Squid is shut down, http traffic loads much faster and without
any noticeable delay.
Where/what else can I look at for the cause of the latency? It becomes
significantly worse during peak use - but as
we're not being choked on bandwidth and things greatly improve when I
shut down squid that narrows it to something
on the server. Is the amount of activity overloading a single squid
process? I'm not seeing any I/O errors in logs and haven't
found any evidence the kernel is under distress.
Any pointers are greatly appreciated.
thanks
-Ryan
