Thank you Amos for clarifying these issues.
I will skip SMP and use single worker since Rock limit my max object
size to 32kb when used in shared environments.
My new cache_dir configuration looks like this now :
cache_dir rock /mnt/ssd/cache/ 300000 max-size=131072
cache_dir aufs /mnt/sas1/cache/store1 1400000 32 512
I have enabled store.log to be used with some other software
collecting data from it
my disks are now mounted with
noatime,barrier=0,journal_async_commit,noauto_da_alloc,nobh,data=writeback,commit=10
I will keep the list posted with my results.
Thanks.
On Sat, Mar 30, 2013 at 12:57 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
> On 30/03/2013 6:33 a.m., Hasanen AL-Bana wrote:
>>
>> Hi,
>>
>> I am running squid 3.2 with an average of 50k req/min. Total received
>> bandwidth is around 200mbit/s.
>> I have problem when my aufs cache_dirs reaches size above 600GB.
>> Traffic starts dropping and going up again , happening every 20~30
>> minutes.
>> I have more that enough RAM in the system (125GB DDR3 !) , All disks
>> are SAS 15k rpm, one of them is and SSD 450GB.
>> So hardware should not cause any problem and I should easily spawn
>> multiple squid workers any time.
>> So what could cause such problems ?
>
>
> #1 - AUFS is *not* SMP-aware component in Squid.
>
> Each of the two workers you are using will be altering the on-disk portion
> of the cache without updating the in-memory index. When the other worker
> encounters these over-written files it will erase them.
>
> For now you are required to use the macros hacks to split the cache_dir
> lines between the workers.
>
>
> #2 - you have multiple cache_dir per disk. Or so it seems from your
> configuration. Is that correct?
>
> * squid load balances between cache_dir treating them as separate physcal
> HDD when it comes to loading calculations.
>
> * the memory requirement for indexing these 1.6 TB of disk space is ~24GB
> per worker plus the 20GB of shared emmory cache ==> 68GB of RAM.
>
>
>
> I advise to you inform Squid about the real disk topology it is working
> with. No more than one AUFS cache_dir per physical disk and allocate the
> cache_dir size to ~80-90% the full available disk space. With each worker
> assigned to use a sub-set of the AUFS cache_dir and associated disks.
>
> For best performance and object de-duplication you should consider using
> rock cache_dir for small objects. They *are* shared between workers, and can
> also share disk space with an AUFS disk.
>
>
>
>> Thank you.
>>
>>
>> ====================================================================================
>> include /etc/squid3/refresh.conf
>>
>>
>> cache_mem 20 GB
>>
>> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
>> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
>> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
>> acl localnet src fc00::/7 # RFC 4193 local private network range
>> acl localnet src fe80::/10 # RFC 4291 link-local (directly
>> plugged) machines
>>
>> acl SSL_ports port 443
>> acl Safe_ports port 80 # http
>> acl Safe_ports port 21 # ftp
>> acl Safe_ports port 443 # https
>> acl Safe_ports port 70 # gopher
>> acl Safe_ports port 210 # wais
>> acl Safe_ports port 1025-65535 # unregistered ports
>> acl Safe_ports port 280 # http-mgmt
>> acl Safe_ports port 488 # gss-http
>> acl Safe_ports port 591 # filemaker
>> acl Safe_ports port 777 # multiling http
>> acl CONNECT method CONNECT
>>
>>
>> http_access allow localhost manager
>> http_access deny manager
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>>
>> http_access allow localnet
>> http_access allow localhost
>>
>> http_access allow all
>
>
> Sure, why bother having security at all?
>
>
>> maximum_object_size_in_memory 512 KB
>> memory_cache_mode always
>> memory_replacement_policy heap GDSF
>> cache_replacement_policy heap LFUDA
>> store_dir_select_algorithm least-load
>> max_open_disk_fds 0
>> maximum_object_size 200 MB
>>
>> cache_swap_high 98
>> cache_swap_low 97
>>
>>
>> # access_log stdio:/var/log/squid3/access.log
>> access_log none
>> cache_log /var/log/squid3/cache.log
>> cache_store_log stdio:/var/log/squid3/store.log
>
>
> Usually the log you want to use is access_log, with the store debug log
> (cache_store_log) disabled.
>
>
>> workers 2
>>
>> cache_dir aufs /mnt/ssd/cache/store1 50000 32 512
>> cache_dir aufs /mnt/ssd/cache/store2 50000 32 512
>> cache_dir aufs /mnt/ssd/cache/store3 50000 32 512
>> cache_dir aufs /mnt/ssd/cache/store4 50000 32 512
>> cache_dir aufs /mnt/ssd/cache/store5 50000 32 512
>> cache_dir aufs /mnt/ssd/cache/store6 50000 32 512
>> cache_dir aufs /mnt/ssd/cache/store7 50000 32 512
>>
>>
>> cache_dir aufs /mnt/sas1/cache/store1 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store2 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store3 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store4 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store5 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store6 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store7 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store8 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store9 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store10 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store11 50000 32 512
>> cache_dir aufs /mnt/sas1/cache/store12 50000 32 512
>>
>>
>> cache_dir aufs /mnt/sas2/cache/store1 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store2 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store3 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store4 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store5 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store6 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store7 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store8 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store9 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store10 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store11 50000 32 512
>> cache_dir aufs /mnt/sas2/cache/store12 50000 32 512
>>
>> ===========================================================
>>
>> and my refresh patterns are :
>>
>> #general
>> refresh_pattern \.(jp(e?g|e|2)|tiff?|bmp|gif|png)
>> 12560 99% 30240 ignore-no-cache
>> ignore-no-store override-expire override-lastmod ignore-private
>> refresh_pattern
>> \.(z(ip|[0-9]{2})|r(ar|[0-9]{2})|jar|bz2|gz|tar|rpm|vpu)
>> 12560 99% 30240 ignore-no-cache ignore-no-store
>> override-expire override-lastmod ignore-private
>> refresh_pattern \.(mp3|wav|og(g|a)|flac|midi?|rm|aac|wma|mka|ape)
>> 12560 99% 30240 ignore-no-cache
>> ignore-no-store override-expire override-lastmod ignore-private
>> refresh_pattern \.(exe|msi|dmg|bin|xpi|iso|mar|psf|cab|ipk|ipa|apk)
>> 12560 99% 30240 ignore-no-cache
>> ignore-no-store override-expire override-lastmod ignore-private
>> refresh_pattern
>>
>> \.(mpeg|ra?m|avi|mp(g|e|4)|mov|divx|asf|wmv|m\dv|rv|vob|asx|ogm|flv|3gp|on2)
>> 12560 99% 30240 ignore-no-cache ignore-no-store override-expire
>> override-lastmod ignore-private
>> refresh_pattern fbcdn 12560 99% 30240 ignore-no-cache
>> ignore-no-store override-expire override-lastmod ignore-private
>> refresh_pattern \.(php|jsp|cgi|asx|html|htm) 12560 100% 30240
>> reload-into-ims ignore-no-cache
>>
>> refresh_pattern -i \.adobe\.com 161280 500% 525948 override-expire
>> ignore-reload
>>
>> #specific sites
>> refresh_pattern \.(ico|video-stats)
>> 12560 100% 30240 override-expire ignore-reload
>> ignore-no-cache ignore-private ignore-auth override-lastmod
>> refresh_pattern ytimg\.com.*\.(jpg|png)
>> 12560 100% 30240 override-expire ignore-reload
>> refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(png|gif)
>> 12560 100% 30240 override-expire ignore-reload
>> ignore-no-cache ignore-private
>> refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)
>> 12560 100% 30240 override-expire ignore-reload
>> refresh_pattern .fbcdn.net.*\.(jpg|gif|png)
>> 12560 100% 30240 ignore-no-cache override-expire
>> ignore-reload
>> refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.
>> 12560 100% 30240 ignore-no-cache ignore-reload
>> override-expire
>> refresh_pattern
>> mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 12560
>> 100% 30240 reload-into-ims override-expire ignore-private
>>
>> refresh_pattern . 1440 100% 2250 reload-into-ims
>
>
> NP: if you have the latest 3.2 releases ignore-no-cache is useless.
>
> Amos
