On 30/03/2013 6:33 a.m., Hasanen AL-Bana wrote:
Hi,
I am running squid 3.2 with an average of 50k req/min. Total received
bandwidth is around 200mbit/s.
I have problem when my aufs cache_dirs reaches size above 600GB.
Traffic starts dropping and going up again , happening every 20~30 minutes.
I have more that enough RAM in the system (125GB DDR3 !) , All disks
are SAS 15k rpm, one of them is and SSD 450GB.
So hardware should not cause any problem and I should easily spawn
multiple squid workers any time.
So what could cause such problems ?
#1 - AUFS is *not* SMP-aware component in Squid.
Each of the two workers you are using will be altering the on-disk
portion of the cache without updating the in-memory index. When the
other worker encounters these over-written files it will erase them.
For now you are required to use the macros hacks to split the cache_dir
lines between the workers.
#2 - you have multiple cache_dir per disk. Or so it seems from your
configuration. Is that correct?
* squid load balances between cache_dir treating them as separate
physcal HDD when it comes to loading calculations.
* the memory requirement for indexing these 1.6 TB of disk space is
~24GB per worker plus the 20GB of shared emmory cache ==> 68GB of RAM.
I advise to you inform Squid about the real disk topology it is working
with. No more than one AUFS cache_dir per physical disk and allocate the
cache_dir size to ~80-90% the full available disk space. With each
worker assigned to use a sub-set of the AUFS cache_dir and associated disks.
For best performance and object de-duplication you should consider using
rock cache_dir for small objects. They *are* shared between workers, and
can also share disk space with an AUFS disk.
Thank you.
====================================================================================
include /etc/squid3/refresh.conf
cache_mem 20 GB
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly
plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow localhost manager
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access allow all
Sure, why bother having security at all?
maximum_object_size_in_memory 512 KB
memory_cache_mode always
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
store_dir_select_algorithm least-load
max_open_disk_fds 0
maximum_object_size 200 MB
cache_swap_high 98
cache_swap_low 97
# access_log stdio:/var/log/squid3/access.log
access_log none
cache_log /var/log/squid3/cache.log
cache_store_log stdio:/var/log/squid3/store.log
Usually the log you want to use is access_log, with the store debug log
(cache_store_log) disabled.
workers 2
cache_dir aufs /mnt/ssd/cache/store1 50000 32 512
cache_dir aufs /mnt/ssd/cache/store2 50000 32 512
cache_dir aufs /mnt/ssd/cache/store3 50000 32 512
cache_dir aufs /mnt/ssd/cache/store4 50000 32 512
cache_dir aufs /mnt/ssd/cache/store5 50000 32 512
cache_dir aufs /mnt/ssd/cache/store6 50000 32 512
cache_dir aufs /mnt/ssd/cache/store7 50000 32 512
cache_dir aufs /mnt/sas1/cache/store1 50000 32 512
cache_dir aufs /mnt/sas1/cache/store2 50000 32 512
cache_dir aufs /mnt/sas1/cache/store3 50000 32 512
cache_dir aufs /mnt/sas1/cache/store4 50000 32 512
cache_dir aufs /mnt/sas1/cache/store5 50000 32 512
cache_dir aufs /mnt/sas1/cache/store6 50000 32 512
cache_dir aufs /mnt/sas1/cache/store7 50000 32 512
cache_dir aufs /mnt/sas1/cache/store8 50000 32 512
cache_dir aufs /mnt/sas1/cache/store9 50000 32 512
cache_dir aufs /mnt/sas1/cache/store10 50000 32 512
cache_dir aufs /mnt/sas1/cache/store11 50000 32 512
cache_dir aufs /mnt/sas1/cache/store12 50000 32 512
cache_dir aufs /mnt/sas2/cache/store1 50000 32 512
cache_dir aufs /mnt/sas2/cache/store2 50000 32 512
cache_dir aufs /mnt/sas2/cache/store3 50000 32 512
cache_dir aufs /mnt/sas2/cache/store4 50000 32 512
cache_dir aufs /mnt/sas2/cache/store5 50000 32 512
cache_dir aufs /mnt/sas2/cache/store6 50000 32 512
cache_dir aufs /mnt/sas2/cache/store7 50000 32 512
cache_dir aufs /mnt/sas2/cache/store8 50000 32 512
cache_dir aufs /mnt/sas2/cache/store9 50000 32 512
cache_dir aufs /mnt/sas2/cache/store10 50000 32 512
cache_dir aufs /mnt/sas2/cache/store11 50000 32 512
cache_dir aufs /mnt/sas2/cache/store12 50000 32 512
===========================================================
and my refresh patterns are :
#general
refresh_pattern \.(jp(e?g|e|2)|tiff?|bmp|gif|png)
12560 99% 30240 ignore-no-cache
ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern
\.(z(ip|[0-9]{2})|r(ar|[0-9]{2})|jar|bz2|gz|tar|rpm|vpu)
12560 99% 30240 ignore-no-cache ignore-no-store
override-expire override-lastmod ignore-private
refresh_pattern \.(mp3|wav|og(g|a)|flac|midi?|rm|aac|wma|mka|ape)
12560 99% 30240 ignore-no-cache
ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern \.(exe|msi|dmg|bin|xpi|iso|mar|psf|cab|ipk|ipa|apk)
12560 99% 30240 ignore-no-cache
ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern
\.(mpeg|ra?m|avi|mp(g|e|4)|mov|divx|asf|wmv|m\dv|rv|vob|asx|ogm|flv|3gp|on2)
12560 99% 30240 ignore-no-cache ignore-no-store override-expire
override-lastmod ignore-private
refresh_pattern fbcdn 12560 99% 30240 ignore-no-cache
ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern \.(php|jsp|cgi|asx|html|htm) 12560 100% 30240
reload-into-ims ignore-no-cache
refresh_pattern -i \.adobe\.com 161280 500% 525948 override-expire
ignore-reload
#specific sites
refresh_pattern \.(ico|video-stats)
12560 100% 30240 override-expire ignore-reload
ignore-no-cache ignore-private ignore-auth override-lastmod
refresh_pattern ytimg\.com.*\.(jpg|png)
12560 100% 30240 override-expire ignore-reload
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(png|gif)
12560 100% 30240 override-expire ignore-reload
ignore-no-cache ignore-private
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)
12560 100% 30240 override-expire ignore-reload
refresh_pattern .fbcdn.net.*\.(jpg|gif|png)
12560 100% 30240 ignore-no-cache override-expire
ignore-reload
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.
12560 100% 30240 ignore-no-cache ignore-reload
override-expire
refresh_pattern
mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 12560
100% 30240 reload-into-ims override-expire ignore-private
refresh_pattern . 1440 100% 2250 reload-into-ims
NP: if you have the latest 3.2 releases ignore-no-cache is useless.
Amos
