Search squid archive

Re: Heavy load squid with high CPU utilization...

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dejan,

Squid is known to be CPU bound under heavy load and the
Quad core running at 1.6 GHz in not the fastest.
A 3.2 GHz dual core will give you double speed.

The config parameter "minimum_object_size 10 KB"
prevents that objects smaller than 10 KB are not written to disk.
I am curious to know why you have this value and if you
benchmarked it, can you share the results ?

The mean object size is 53 KB and the parameter
maximum_object_size_in_memory 50 KB
implies that you have a relatively large number of hot objects
that do not stay in memory.
The memory hit % is low and the disk hit % is high, so the
maximum_object_size_in_memory should be increased.
I suggest 96 KB, monitor the memory hit % and increase more
if necessary.

client_persistent_connections and server_persistent_connections
are off.  The default is on and usually gives better performance.
Why are they off ?

The TCP window scaling is off.  This is a performance penalty
for large objects since it uses the select/epoll loop a lot more
because objects arrive in more smaller pieces.
Why is it off ?
If you have a good reason to set it off I recommend to use
the maximum size for fixed TCP window size: 64K (squid parameter
tcp_recv_bufsize) to reduce the number of calls to select/epoll.

You use one disk solely for cache.  This can be better
if you use a battery-backed disk I/O controller with
256MB cache.
And the obvious: more disks is good for overall performance

Marcus

Dejan Zivanic wrote:
Regards,

we have heavy load (over 6k requests per minute) intercepting squid loading about 70-80Mbps traffic. I have notices that CPU usage of squid process never goes down from 50% and usually goes up to over 90%.

We plan to upgrade to 120Mbps link and this can be major problem if we cannot solve it.
Every suggestions will be appreciated...

Best regards,
Zivanic Dejan

Server: FujitsuSiemens Primergy Econel 200 S2 (Quad Xeon E5310 on 1.6Ghz) with 6GB ECC ram.
Storage: 2xSata udma6 250GB
OS: Fedora14 x64

OS is installed on sda1, sda2 is only for cache_dir.

Squid Cache: Version 3.1.11
configure options: '--prefix=/usr' '--localstatedir=/var' '--sysconfdir=/etc/squid' '--with-filedescriptors=16384' '--enable-removal-policies=heap,lru' '--enable-delay-pools' '--enable-epoll' '--enable-stopreio=ufs,aufs,diskd' '--enable-async-io' '--with-pthreads' '--disable-dlmalloc' '--with-large-files' '--enable-htcp' '--enable-large-cache-files' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-dl' '--enable-ltdl-convenience' '--enable-linux-netfilter' --with-squid=/root/install/squid-3.1.11

Linux 2.6.35.6-45.fc14.x86_64 #1 SMP Mon Oct 18 23:57:44 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
squid]# uptime
20:27:36 up 4 days, 22:07,  3 users,  load average: 0.93, 0.84, 0.85
squid]# free
             total       used       free     shared    buffers     cached
Mem:       6125216    4956620    1168596          0     377916    2568624
-/+ buffers/cache:    2010080    4115136
Swap:      4194296       9824    4184472

Squid Object Cache: Version 3.1.11
Start Time:    Mon, 21 Mar 2011 12:42:30 GMT
Current Time:    Tue, 22 Mar 2011 19:19:01 GMT
Connection information for squid:
    Number of clients accessing cache:    999
    Number of HTTP requests received:    11471323
    Number of ICP messages received:    0
    Number of ICP messages sent:    0
    Number of queued ICP replies:    0
    Number of HTCP messages received:    0
    Number of HTCP messages sent:    0
    Request failure ratio:     0.00
    Average HTTP requests per minute since start:    6246.2
    Average ICP messages per minute since start:    0.0
    Select loop called: 320350087 times, 0.344 ms avg
Cache information for squid:
    Hits as % of all requests:    5min: 10.0%, 60min: 9.4%
    Hits as % of bytes sent:    5min: 12.2%, 60min: 11.9%
    Memory hits as % of hit requests:    5min: 11.5%, 60min: 10.3%
    Disk hits as % of hit requests:    5min: 60.4%, 60min: 59.5%
    Storage Swap size:    72469364 KB
    Storage Swap capacity:    39.3% used, 60.7% free
    Storage Mem size:    1038552 KB
    Storage Mem capacity:    100.0% used,  0.0% free
    Mean Object Size:    53.58 KB
    Requests given to unlinkd:    0
Median Service Times (seconds)  5 min    60 min:
    HTTP Requests (All):   0.22004  0.23230
    Cache Misses:          0.27332  0.25890
    Cache Hits:            0.01648  0.01745
    Near Hits:             0.25890  0.18699
    Not-Modified Replies:  0.00179  0.00179
    DNS Lookups:           0.07618  0.08334
    ICP Queries:           0.00000  0.00000
Resource usage for squid:
    UP Time:    110191.361 seconds
    CPU Time:    41856.331 seconds
    CPU Usage:    60.99%
    CPU Usage, 5 minute avg:    75.64%
    CPU Usage, 60 minute avg:    81.66%
    Process Data Segment Size via sbrk(): 1474024 KB
    Maximum Resident Size: 5983984 KB
    Page faults with physical i/o: 13
Memory usage for squid via mallinfo():
    Total space in arena:  1474156 KB
    Ordinary blocks:       1467177 KB  10620 blks
    Small blocks:               0 KB      0 blks
    Holding blocks:         24216 KB      9 blks
    Free Small blocks:          0 KB
    Free Ordinary blocks:    6978 KB
    Total in use:          1491393 KB 100%
    Total free:              6978 KB 0%
    Total size:            1498372 KB
Memory accounted for:
    Total accounted:       1332348 KB  89%
    memPool accounted:     1332348 KB  89%
    memPool unaccounted:   166023 KB  11%
    memPoolAlloc calls: 2768926328
    memPoolFree calls:  2896226918
File descriptor usage for squid:
    Maximum number of file descriptors:   16384
    Largest file desc currently in use:   1069
    Number of file desc currently in use:  867
    Files queued for open:                   0
    Available number of file descriptors: 15517
    Reserved number of file descriptors:   100
    Store Disk files open:                   8
Internal Data Structures:
    1364435 StoreEntries
     65986 StoreEntries with MemObjects
     65640 Hot Object Cache Items
    1352638 on-disk objects

sysctl.conf:
//
fs.file-max = 360000
vm.drop_caches = 3
vm.swappiness = 3
net.ipv4.ip_local_port_range = 2048 65000
net.core.rmem_max = 16777216
net.core.wmem_max = 32777216
net.ipv4.tcp_low_latency = 1
net.core.netdev_max_backlog = 4000
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
vm.min_free_kbytes = 70000
net.core.somaxconn = 65536
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.netfilter.nf_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_buckets = 1048576
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 3
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 100
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_icmp_timeout = 10
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_udp_timeout = 10
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_established = 21600
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 5
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 5
net.netfilter.nf_conntrack_checksum = 1
//

squid.conf:
//
http_port 5555 intercept
icp_port 0
maximum_icp_query_timeout 3000
mcast_icp_query_timeout 2000
dead_peer_timeout 10 seconds
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_mem 1024 MB
cache_swap_low 98
cache_swap_high 99
maximum_object_size 250 MB
minimum_object_size 10 KB
maximum_object_size_in_memory 50 KB
ipcache_size 16384
ipcache_low 90
ipcache_high 95
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
cache_dir aufs /cache00 180000 32 256
cache_access_log /var/log/squid/access.log common
cache_log /var/log/squid/cache.log
cache_store_log none
cache_swap_log /var/spool/squid/cache_swap_log
log_ip_on_direct off
pid_filename /var/run/squid.pid
debug_options ALL,1
dns_timeout 20 seconds
dns_nameservers 212.200.191.150 212.200.113.130
auth_param basic children 15
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off
refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern .        0    20%    4320
forward_timeout 20 seconds
connect_timeout 10 seconds
peer_connect_timeout 10 seconds
read_timeout 30 seconds
request_timeout 10 seconds
persistent_request_timeout 10 seconds
client_lifetime 360 minutes
half_closed_clients off
pconn_timeout 10 seconds
shutdown_lifetime 10 seconds
acl manager proto cache_object
acl localhost src 127.0.0.1
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443 563    # https, snews
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny opstina2 !vreme
http_access allow kladovonet
http_access allow kladovonet-public
http_access deny all
tcp_outgoing_address 172.29.203.200
cache_peer 172.29.202.252 parent 8080 7 name=dsl1 round-robin no-query no-digest weight=2 cache_peer 172.29.202.253 parent 8080 7 name=dsl2 round-robin no-query no-digest weight=2 cache_peer 172.29.202.251 parent 8080 7 name=dsl3 round-robin no-query no-digest weight=2
acl kroz-adsl url_regex -i "/etc/adsl"
cache_peer_access adsl1 allow kroz-adsl
cache_peer_access adsl2 allow kroz-adsl
cache_peer_access adsl3 allow kroz-adsl
never_direct allow kroz-adsl
http_reply_access allow all
icp_access deny all
cache_mgr Zivanic_Dejan_[zivanicd@xxxxxxxxxxxxxx]
cache_effective_user squid
logfile_rotate 10
memory_pools off
error_directory /usr/share/errors/sr-latn
ignore_unknown_nameservers off
client_persistent_connections off
server_persistent_connections off
//






[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux