Thanks for the response, Adrian.
Is recompile required to change to internal DNS?
I've disabled ECN, pmtu_disc and mtu_probing.
cache_dir is as follows:
(recommended by Henrik)
cache_dir aufs /squid0 125000 128 256
cache_dir aufs /squid1 125000 128 256
cache_dir aufs /squid2 125000 128 256
cache_dir aufs /squid3 125000 128 256
cache_dir aufs /squid4 125000 128 256
cache_dir aufs /squid5 125000 128 256
cache_dir aufs /squid6 125000 128 256
cache_dir aufs /squid7 125000 128 256
No peak data available, here's some pre-peak data:
Cache Manager menu
5-MINUTE AVERAGE
sample_start_time = 1222199580.85434 (Tue, 23 Sep 2008 19:53:00 GMT)
sample_end_time = 1222199905.507274 (Tue, 23 Sep 2008 19:58:25 GMT)
client_http.requests = 268.239526/sec
client_http.hits = 111.741117/sec
client_http.errors = 0.000000/sec
IOSTAT shows lots of idle time - I'm unclear what you mean by
"profiling" ?
Also, have not tried running w/out any cache - can you explain
how this is done?
appreciate the assistance.
-Ryan
Adrian Chadd wrote:
Firstly, you should use the internal DNS code instead of the external
DNS helpers.
Secondly, I'd do a little debugging to see if its network related -
make sure you've disabled PMTU for example, as WCCP doesn't redirect
the ICMP needed. Other things like Window scaling negotiation and such
may contribute.
From a server side of things, what cache_dir config are you using?
Whats your average/peak request rate? What about disk IO? Have you
done any profiling? Have you tried running the proxy without any disk
cache to see if the problem goes away?
~ terabyte of cache is quite large; I don't think any developers have
a terabyte of storage in a box this size in a testing environment.
2008/9/24 Ryan Goddard <rgoddard@xxxxxxxxxxxx>:
Squid 2.7.STABLE1-20080528 on Debian Linux 2.6.19.7
running on quad dual-core 2.6mhz Opterons with 32 gig RAM; 8x140GB disk
partitions
using WCCP L2 redirects transparently from a Cisco 4948 GigE switch
Server has one GigE NIC for the incoming redirects and two GigE NICs for
outbound http requests.
Using IPTables to port forward HTTP to Squid; no ICP, auth, etc.; strictly a
web cache using heap/LFUDA replacement
and 16GB memory allocated with mem pools on, no limit.
Used in an ISP environment, accommodating approx. 8k predominately cable
modem customers during peak.
Issue we're experiencing is some web pages taking in excess of 20 seconds to
load, marked latency for customers
running web-based speed tests, etc.
Cache.log and Access.log aren't indicating any errors or timeouts; system
operates 96 DNS instances and 32k file descriptors
(neither has gotten maxed yet).
General Runtime Info from Cachemgr taken during pre-peak usage:
Start Time: Tue, 23 Sep 2008 18:07:37 GMT
Current Time: Tue, 23 Sep 2008 21:00:49 GMT
Connection information for squid:
Number of clients accessing cache: 3382
Number of HTTP requests received: 2331742
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 13463.4
Average ICP messages per minute since start: 0.0
Select loop called: 11255153 times, 0.923 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 42.6%, 60min: 40.0%
Byte Hit Ratios: 5min: 21.2%, 60min: 18.6%
Request Memory Hit Ratios: 5min: 18.3%, 60min: 17.2%
Request Disk Hit Ratios: 5min: 33.6%, 60min: 33.3%
Storage Swap size: 952545580 KB
Storage Mem size: 8237648 KB
Mean Object Size: 40.43 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.19742 0.12106
Cache Misses: 0.27332 0.17711
Cache Hits: 0.08265 0.03622
Near Hits: 0.27332 0.16775
Not-Modified Replies: 0.02317 0.00865
DNS Lookups: 0.09535 0.04854
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 10391.501 seconds
CPU Time: 4708.150 seconds
CPU Usage: 45.31%
CPU Usage, 5 minute avg: 33.29%
CPU Usage, 60 minute avg: 33.36%
Process Data Segment Size via sbrk(): 1041332 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 4
Memory usage for squid via mallinfo():
Total space in arena: 373684 KB
Ordinary blocks: 372642 KB 809 blks
Small blocks: 0 KB 0 blks
Holding blocks: 216088 KB 21 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 1041 KB
Total in use: 588730 KB 100%
Total free: 1041 KB 0%
Total size: 589772 KB
Memory accounted for:
Total accounted: 11355185 KB
memPoolAlloc calls: 439418241
memPoolFree calls: 378603777
File descriptor usage for squid:
Maximum number of file descriptors: 32000
Largest file desc currently in use: 9171
Number of file desc currently in use: 8112
Files queued for open: 2
Available number of file descriptors: 23886
Reserved number of file descriptors: 100
Store Disk files open: 175
IO loop method: epoll
Internal Data Structures:
23570637 StoreEntries
532260 StoreEntries with MemObjects
531496 Hot Object Cache Items
23561001 on-disk objects
Generated Tue, 23 Sep 2008 21:00:47 GMT, by
cachemgr.cgi/2.7.STABLE1-20080528@xxxxxxxxxxxxxxxxxx
TCPDUMP shows packets traversing all interfaces as expected; bandwidth to
both upstream providers isn't being maxed
and when Squid is shut down, http traffic loads much faster and without any
noticeable delay.
Where/what else can I look at for the cause of the latency? It becomes
significantly worse during peak use - but as
we're not being choked on bandwidth and things greatly improve when I shut
down squid that narrows it to something
on the server. Is the amount of activity overloading a single squid
process? I'm not seeing any I/O errors in logs and haven't
found any evidence the kernel is under distress.
Any pointers are greatly appreciated.
thanks
-Ryan
