Dejan,
Squid is known to be CPU bound under heavy load and the
Quad core running at 1.6 GHz in not the fastest.
A 3.2 GHz dual core will give you double speed.
The config parameter "minimum_object_size 10 KB"
prevents that objects smaller than 10 KB are not written to disk.
I am curious to know why you have this value and if you
benchmarked it, can you share the results ?
The mean object size is 53 KB and the parameter
maximum_object_size_in_memory 50 KB
implies that you have a relatively large number of hot objects
that do not stay in memory.
The memory hit % is low and the disk hit % is high, so the
maximum_object_size_in_memory should be increased.
I suggest 96 KB, monitor the memory hit % and increase more
if necessary.
client_persistent_connections and server_persistent_connections
are off. The default is on and usually gives better performance.
Why are they off ?
The TCP window scaling is off. This is a performance penalty
for large objects since it uses the select/epoll loop a lot more
because objects arrive in more smaller pieces.
Why is it off ?
If you have a good reason to set it off I recommend to use
the maximum size for fixed TCP window size: 64K (squid parameter
tcp_recv_bufsize) to reduce the number of calls to select/epoll.
You use one disk solely for cache. This can be better
if you use a battery-backed disk I/O controller with
256MB cache.
And the obvious: more disks is good for overall performance
Marcus
Dejan Zivanic wrote:
Regards,
we have heavy load (over 6k requests per minute) intercepting squid
loading about 70-80Mbps traffic.
I have notices that CPU usage of squid process never goes down from 50%
and usually goes up to over 90%.
We plan to upgrade to 120Mbps link and this can be major problem if we
cannot solve it.
Every suggestions will be appreciated...
Best regards,
Zivanic Dejan
Server: FujitsuSiemens Primergy Econel 200 S2 (Quad Xeon E5310 on
1.6Ghz) with 6GB ECC ram.
Storage: 2xSata udma6 250GB
OS: Fedora14 x64
OS is installed on sda1, sda2 is only for cache_dir.
Squid Cache: Version 3.1.11
configure options: '--prefix=/usr' '--localstatedir=/var'
'--sysconfdir=/etc/squid' '--with-filedescriptors=16384'
'--enable-removal-policies=heap,lru' '--enable-delay-pools'
'--enable-epoll' '--enable-stopreio=ufs,aufs,diskd' '--enable-async-io'
'--with-pthreads' '--disable-dlmalloc' '--with-large-files'
'--enable-htcp' '--enable-large-cache-files' '--enable-wccpv2'
'--enable-esi' '--with-aio' '--with-dl' '--enable-ltdl-convenience'
'--enable-linux-netfilter' --with-squid=/root/install/squid-3.1.11
Linux 2.6.35.6-45.fc14.x86_64 #1 SMP Mon Oct 18 23:57:44 UTC 2010 x86_64
x86_64 x86_64 GNU/Linux
squid]# uptime
20:27:36 up 4 days, 22:07, 3 users, load average: 0.93, 0.84, 0.85
squid]# free
total used free shared buffers cached
Mem: 6125216 4956620 1168596 0 377916 2568624
-/+ buffers/cache: 2010080 4115136
Swap: 4194296 9824 4184472
Squid Object Cache: Version 3.1.11
Start Time: Mon, 21 Mar 2011 12:42:30 GMT
Current Time: Tue, 22 Mar 2011 19:19:01 GMT
Connection information for squid:
Number of clients accessing cache: 999
Number of HTTP requests received: 11471323
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 6246.2
Average ICP messages per minute since start: 0.0
Select loop called: 320350087 times, 0.344 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 10.0%, 60min: 9.4%
Hits as % of bytes sent: 5min: 12.2%, 60min: 11.9%
Memory hits as % of hit requests: 5min: 11.5%, 60min: 10.3%
Disk hits as % of hit requests: 5min: 60.4%, 60min: 59.5%
Storage Swap size: 72469364 KB
Storage Swap capacity: 39.3% used, 60.7% free
Storage Mem size: 1038552 KB
Storage Mem capacity: 100.0% used, 0.0% free
Mean Object Size: 53.58 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.22004 0.23230
Cache Misses: 0.27332 0.25890
Cache Hits: 0.01648 0.01745
Near Hits: 0.25890 0.18699
Not-Modified Replies: 0.00179 0.00179
DNS Lookups: 0.07618 0.08334
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 110191.361 seconds
CPU Time: 41856.331 seconds
CPU Usage: 60.99%
CPU Usage, 5 minute avg: 75.64%
CPU Usage, 60 minute avg: 81.66%
Process Data Segment Size via sbrk(): 1474024 KB
Maximum Resident Size: 5983984 KB
Page faults with physical i/o: 13
Memory usage for squid via mallinfo():
Total space in arena: 1474156 KB
Ordinary blocks: 1467177 KB 10620 blks
Small blocks: 0 KB 0 blks
Holding blocks: 24216 KB 9 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 6978 KB
Total in use: 1491393 KB 100%
Total free: 6978 KB 0%
Total size: 1498372 KB
Memory accounted for:
Total accounted: 1332348 KB 89%
memPool accounted: 1332348 KB 89%
memPool unaccounted: 166023 KB 11%
memPoolAlloc calls: 2768926328
memPoolFree calls: 2896226918
File descriptor usage for squid:
Maximum number of file descriptors: 16384
Largest file desc currently in use: 1069
Number of file desc currently in use: 867
Files queued for open: 0
Available number of file descriptors: 15517
Reserved number of file descriptors: 100
Store Disk files open: 8
Internal Data Structures:
1364435 StoreEntries
65986 StoreEntries with MemObjects
65640 Hot Object Cache Items
1352638 on-disk objects
sysctl.conf:
//
fs.file-max = 360000
vm.drop_caches = 3
vm.swappiness = 3
net.ipv4.ip_local_port_range = 2048 65000
net.core.rmem_max = 16777216
net.core.wmem_max = 32777216
net.ipv4.tcp_low_latency = 1
net.core.netdev_max_backlog = 4000
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
vm.min_free_kbytes = 70000
net.core.somaxconn = 65536
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.netfilter.nf_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_buckets = 1048576
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 3
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 100
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_icmp_timeout = 10
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_udp_timeout = 10
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_established = 21600
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 5
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 5
net.netfilter.nf_conntrack_checksum = 1
//
squid.conf:
//
http_port 5555 intercept
icp_port 0
maximum_icp_query_timeout 3000
mcast_icp_query_timeout 2000
dead_peer_timeout 10 seconds
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_mem 1024 MB
cache_swap_low 98
cache_swap_high 99
maximum_object_size 250 MB
minimum_object_size 10 KB
maximum_object_size_in_memory 50 KB
ipcache_size 16384
ipcache_low 90
ipcache_high 95
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
cache_dir aufs /cache00 180000 32 256
cache_access_log /var/log/squid/access.log common
cache_log /var/log/squid/cache.log
cache_store_log none
cache_swap_log /var/spool/squid/cache_swap_log
log_ip_on_direct off
pid_filename /var/run/squid.pid
debug_options ALL,1
dns_timeout 20 seconds
dns_nameservers 212.200.191.150 212.200.113.130
auth_param basic children 15
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
forward_timeout 20 seconds
connect_timeout 10 seconds
peer_connect_timeout 10 seconds
read_timeout 30 seconds
request_timeout 10 seconds
persistent_request_timeout 10 seconds
client_lifetime 360 minutes
half_closed_clients off
pconn_timeout 10 seconds
shutdown_lifetime 10 seconds
acl manager proto cache_object
acl localhost src 127.0.0.1
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny opstina2 !vreme
http_access allow kladovonet
http_access allow kladovonet-public
http_access deny all
tcp_outgoing_address 172.29.203.200
cache_peer 172.29.202.252 parent 8080 7 name=dsl1 round-robin no-query
no-digest weight=2
cache_peer 172.29.202.253 parent 8080 7 name=dsl2 round-robin no-query
no-digest weight=2
cache_peer 172.29.202.251 parent 8080 7 name=dsl3 round-robin no-query
no-digest weight=2
acl kroz-adsl url_regex -i "/etc/adsl"
cache_peer_access adsl1 allow kroz-adsl
cache_peer_access adsl2 allow kroz-adsl
cache_peer_access adsl3 allow kroz-adsl
never_direct allow kroz-adsl
http_reply_access allow all
icp_access deny all
cache_mgr Zivanic_Dejan_[zivanicd@xxxxxxxxxxxxxx]
cache_effective_user squid
logfile_rotate 10
memory_pools off
error_directory /usr/share/errors/sr-latn
ignore_unknown_nameservers off
client_persistent_connections off
server_persistent_connections off
//
