Search squid archive

Re: Squid losing connectivity for 30 seconds

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2/12/2011 3:16 a.m., Elie Merhej wrote:


 Hi,

I am currently facing a problem that I wasn't able to find a solution for in the mailing list or on the internet, My squid is dying for 30 seconds every one hour at the same exact time, squid process will still be running, I lose my wccp connectivity, the cache peers detect the squid as a dead sibling, and the squid cannot server any requests The network connectivity of the sever is not affected (a ping to the squid's ip doesn't timeout)

The problem doesn't start immediately when the squid is installed on the server (The server is dedicated as a squid)
It starts when the cache directories starts to fill up,
I have started my setup with 10 cache directors, the squid will start having the problem when the cache directories are above 50% filled when i change the number of cache directory (9,8,...) the squid works for a while then the same problem
cache_dir aufs /cache1/squid 90000 140 256
cache_dir aufs /cache2/squid 90000 140 256
cache_dir aufs /cache3/squid 90000 140 256
cache_dir aufs /cache4/squid 90000 140 256
cache_dir aufs /cache5/squid 90000 140 256
cache_dir aufs /cache6/squid 90000 140 256
cache_dir aufs /cache7/squid 90000 140 256
cache_dir aufs /cache8/squid 90000 140 256
cache_dir aufs /cache9/squid 90000 140 256
cache_dir aufs /cache10/squid 80000 140 256

I have 1 terabyte of storage
Finally I created two cache dircetories (One on each HDD) but the problem persisted

You have 2 HDD?  but, but, you have 10 cache_dir.
We repeatedly say "one cache_dir per disk" or similar. In particular one cache_dir per physical drive spindle (for "disks" made up of multiple physical spindles) wherever possible with physical drives/spindles mounting separately to ensure the pairing. Squid performs a very unusual pattern of disk I/O which stress them down to the hardware controller level and make this kind of detail critical for anything like good speed. Avoiding cache_dir object limitations by adding more UFS-based dirs to one disk does not improve the situation.

That is a problem which will be affecting your Squid all the time though, possibly making the source of the pause worse.

From teh description I believe it is garbage collection on the cache directories. The pauses can be visible when garbage collecting any caches over a few dozen GB. The squid default "swap_high" and "swap_low" values are "5" apart, with at minimum being a value of 0 apart. These are whole % points of the total cache size, being erased from disk in a somewhat random-access style across the cache area. I did mention uncommon disk I/O patterns, right?

To be sure what it is, you can use the "strace" tool to the squid worker process (the second PID in current stable Squids) and see what is running. But given the hourly regularity and past experience with others on similar cache sizes, I'm almost certain its the garbage collection.

Amos


Hi Amos,

Thank you for your fast reply,
I have 2 HDD (450GB and 600GB)
df -h displays that i have 357Gb and 505GB available
In my last test, my cache dir where:
cache_swap_low 90
cache_swap_high 95

This is not. For anything more than 10-20 GB I recommend setting it to no more than 1 apart, possibly the same value if that works. Squid has a light but CPU-intensive and possibly long garbage removal cycle above cache_swap_low, and a much more aggressive but faster and less CPU intensive removal above cache_swap_high. On large caches it is better in terms of downtime going straight to the aggressive removal and clearing disk space fast, despite the bandwidth cost replacing any items the light removal would have left.


Amos

Hi Amos,

I have changed the swap_high 90 and swap_low 90 with two cache dir (one for each HDD), i still have the same problem,
I did an strace (when the problem occured)
------ ----------- ----------- --------- --------- ----------------
 23.06    0.004769           0     85681        96 write
 21.07    0.004359           0     24658         5 futex
 19.34    0.004001         800         5           open
  6.54    0.001352           0      5101      5101 connect
  6.46    0.001337           3       491           epoll_wait
  5.34    0.001104           0     51938      9453 read
  3.90    0.000806           0     39727           close
  3.54    0.000733           0     86400           epoll_ctl
  3.54    0.000732           0     32357           sendto
  2.02    0.000417           0     56721           recvmsg
  1.84    0.000381           0     24064           socket
  0.96    0.000199           0     56264           fcntl
  0.77    0.000159           0      6366       329 accept
  0.53    0.000109           0     24033           bind
  0.52    0.000108           0     30085           getsockname
  0.21    0.000044           0     11200           stat
  0.21    0.000044           0      6998       359 recvfrom
  0.09    0.000019           0      5085           getsockopt
  0.06    0.000012           0      2887           lseek
  0.00    0.000000           0        98           brk
  0.00    0.000000           0        16           dup2
  0.00    0.000000           0     10314           setsockopt
  0.00    0.000000           0         4           getdents
  0.00    0.000000           0         3           getrusage
------ ----------- ----------- --------- --------- ----------------
100.00    0.020685                560496     15343 total

this is the strace of squid when it is working normally:
------ ----------- ----------- --------- --------- ----------------
 24.88    0.015887           0    455793       169 write
 13.72    0.008764           0    112185           epoll_wait
 11.67    0.007454           0    256256     27158 read
  8.47    0.005408           0    169133           sendto
  6.94    0.004430           0    159596           close
  6.85    0.004373           0    387359           epoll_ctl
  6.42    0.004102           0     19651     19651 connect
  5.54    0.003538           0    290289           recvmsg
  3.81    0.002431           0    116515           socket
  3.53    0.002254           0    164750           futex
  1.68    0.001075           0    207688           fcntl
  1.53    0.000974           0     95228     23139 recvfrom
  1.29    0.000821           0     33408     12259 accept
  1.14    0.000726           0     46582           stat
  1.11    0.000707           0    110826           bind
  0.85    0.000544           0    137574           getsockname
  0.32    0.000204           0     21642           getsockopt
  0.26    0.000165           0     39502           setsockopt
  0.01    0.000007           0      8092           lseek
  0.00    0.000000           0       248           open
  0.00    0.000000           0         4           brk
  0.00    0.000000           0        88           dup2
  0.00    0.000000           0        14           getdents
  0.00    0.000000           0         6           getrusage
------ ----------- ----------- --------- --------- ----------------
100.00    0.063864               2832429     82376 total

Do you have any suggestions to solve the issue, can I run the garbage collector more frequently, is it better to change the cache_dir type from aufs to something else?
Do you see the problem in the strace?

Thank you,
Elie


Hi,

Please note that squid is facing the same problem even when their is no activity or any clients connected to it

Regards
Elie

Hi,

here is the strace result
-----------------------------------------------------------------------------------------------------
<snip looks perfectly normal traffic, file opening and closing data reading, DNS lookups and other network read/writes>
read(165, "!", 256)                     = 1
<snip bunch of other normal traffic>

read(165, "!", 256)                     = 1
----------------------------------------------------------------------------------------------------------------------------------------------------
Squid is freezing at this point

The 1-byte read on FD #165 seems odd. Particularly suspicious being just before a pause and only having a constant 256 byte buffer space available. No ideas what it is yet though.


Here is my compilation options
-------------------------------------------------------------------------------------------------------------------------------------------------------- ./configure '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,negotiate' '--enable-external-acl-helpers=ip_user,unix_group,wbinfo_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,ufs' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=proxy' '--with-filedescriptors=65536' '--with-dl' '--with-pthreads' '--with-libcap' '--with-netfilter-conntrack' '--with-openssl' '--enable-inline' '--enable-uselect' '--enable-disk-io' '--disable-htcp' '--with-gnu-ld' '--with-build-environment=default' '--enable-carp' '--enable-async-io=26' --with-squid=/home/squid-3.1.15 --enable-ltdl-convenience ---------------------------------------------------------------------------------------------------------------------------------------------------------
Here is my squid.conf:

Can't help myself, I digress into a config audit... completely off-topic.

---------------------------------------------------------------------------------------------------------------------------------------------------------
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
acl clients src x.x.x.x
#icp acl
acl squidFarm src x.x.x.x
acl self src x.x.x.x

#ICAP acl
acl icap_port1 myportname 3144
acl icap_port2 myportname 3145
acl icap_port3 myportname 3146
acl icap_port5 myportname 3148

http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
#prevent digest loop
http_access deny self
http_access allow clients
http_access deny all
http_reply_access allow all
#icp_access allow all
icp_port 3130
icp_access allow squidFarm
icp_access deny all

http_port 3129 tproxy
http_port 3128 transparent
http_port 3144 tproxy
http_port 3146 tproxy
http_port 3145 tproxy
http_port 3148 tproxy

forwarded_for off
via off
visible_hostname x.x.x.x
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid
# Image files
refresh_pattern -i \.png$                10080   90%     43200
refresh_pattern -i \.gif$                10080   90%     43200
refresh_pattern -i \.jpg$                10080   90%     43200
refresh_pattern -i \.jpeg$               10080   90%     43200
refresh_pattern -i \.bmp$                10080   90%     43200
refresh_pattern -i \.tif$                10080   90%     43200
refresh_pattern -i \.tiff$               10080   90%     43200

This is *the* most inefficient way to do this. The refresh_pattern set is tested for every single cached object load. On top of that each pattern line is an individual regex pettern, which is almost the slowest match type Squid can perform. You will gain in proxy performance by collapsing these regex patterns down into one line. Like so:

   refresh_pattern -i \.(png|gif|jpe?g|bmp|tiff?)$  10080 90% 43200

same for the others...

# Compressed files
refresh_pattern -i \.zip$                10080   90%     43200
refresh_pattern -i \.rar$                10080   90%     43200
refresh_pattern -i \.tar$                10080   90%     43200
refresh_pattern -i \.gz$                 10080   90%     43200
refresh_pattern -i \.tgz$                10080   90%     43200
refresh_pattern -i \.z$                  10080   90%     43200
refresh_pattern -i \.arj$                10080   90%     43200
refresh_pattern -i \.lha$                10080   90%     43200
refresh_pattern -i \.lzh$                10080   90%     43200

# Binary files
refresh_pattern -i \.exe$                10080   90%     43200
refresh_pattern -i \.msi$                10080   90%     43200

# Multimedia files
refresh_pattern -i \.mp3$                10080   90%     43200
refresh_pattern -i \.wav$                10080   90%     43200
refresh_pattern -i \.mid$                10080   90%     43200
refresh_pattern -i \.midi$               10080   90%     43200
refresh_pattern -i \.ram$                10080   90%     43200
refresh_pattern -i \.ra$                 10080   90%     43200
refresh_pattern -i \.mov$                10080   90%     43200
refresh_pattern -i \.avi$                10080   90%     43200
refresh_pattern -i \.wmv$                10080   90%     43200
refresh_pattern -i \.mpg$                10080   90%     43200
refresh_pattern -i \.mpg$                10080   90%     43200
refresh_pattern -i \.mpg$                10080   90%     43200
refresh_pattern -i \.mpeg$               10080   90%     43200
refresh_pattern -i \.swf$                10080   90%     43200

# Document files
refresh_pattern -i \.pdf$                10080   90%     43200
refresh_pattern -i \.ps$                 10080   90%     43200
refresh_pattern -i \.doc$                10080   90%     43200
refresh_pattern -i \.ppt$                10080   90%     43200
refresh_pattern -i \.pps$                10080   90%     43200
#windows update refresh paterns
refresh_pattern windowsupdate.com/.*\.(cab|exe|psf) 4320 100% 43200 reload-into-ims refresh_pattern download.microsoft.com/.*\.(cab|exe|psf) 4320 100% 43200 reload-into-ims refresh_pattern armdl.adobe.com/.*\.(cab|msp|msi) 4320 100% 43200 reload-into-ims
#default refresh paterns
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

client_db on
cache_mem 256 MB
cache_swap_low 90
cache_swap_high 90
maximum_object_size 512 MB
maximum_object_size_in_memory 20 KB
cache_dir aufs /cache1/squid 320000 480 256
cache_dir aufs /cache2/squid 480000 700 256

#logformat modified %tl Request time:%tr Status:%Ss/%03>Hs Client:%>a URL:%ru Server:%<A Type:%mt
logformat modified %tl %>a %ru %<A
access_log /var/log/squid/access.log modified
cache_log /var/log/squid/cache.log
cache_store_log none
coredump_dir /var/spool/squid
half_closed_clients off

snmp_port x
acl cacti src x.x.x.x
acl snmpcommunity snmp_community xxxx
snmp_access allow snmpcommunity xxxx
snmp_access allow snmpcommunity localhost
snmp_access deny all

wccp2_router x.x.x.x
wccp2_forwarding_method l2
wccp2_return_method l2
wccp2_service dynamic x
wccp2_service_info x protocol=tcp flags=src_ip_hash priority=240 ports=80
wccp2_service dynamic x
wccp2_service_info x protocol=tcp flags=dst_ip_hash,ports_source priority=240 ports=80
wccp2_assignment_method mask


#icp configuration
maximum_icp_query_timeout 30
cache_peer x.x.x.x sibling 3128 3130 proxy-only no-tproxy
cache_peer x.x.x.x sibling 3128 3130 proxy-only no-tproxy
cache_peer x.x.x.x sibling 3128 3130 proxy-only no-tproxy
log_icp_queries off
miss_access allow squidFarm
miss_access deny all

So if I understand this right. You have a layer of proxies defined as "squidFarm" which client traffic MUST pass through *first* before they are allowed to fetch MISS requests from this proxy. Yet you are receiving WCCP traffic directly at this proxy with both NAT and TPROXY?

This miss_access policy seems decidedly odd. Perhapse you can enlighten me?




# ICAP configuration
icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Client-Username
icap_preview_enable on
icap_preview_size 1024



logformat squid %tl %icap::tt %icap::tr %>a %icap::rm %icap::ru
icap_log /var/log/squid/icap.log squid
icap_service service_req reqmod_precache bypass=1 icap://x.x.x.x:x/reqmod
adaptation_access service_req allow icap_port1
icap_service service_req_2 reqmod_precache bypass=1 icap://x.x.x.x:x/reqmod
adaptation_access service_req_2 allow icap_port2
icap_service service_req_3 reqmod_precache bypass=1 icap://x.x.x.x:x/reqmod
adaptation_access service_req_3 allow icap_port3
icap_service service_req_5 reqmod_precache bypass=1 icap://x.x.x.x:x/reqmod
adaptation_access service_req_5 allow icap_port5
---------------------------------------------------------------------------------------------------------------------------------------------------------

Please advise,
Thank you
Elie



[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux