Does this look reasonable? I still have the analysis to start with after this point but will use some linux tools to help with that... auth_param basic realm P*****r ProxyServer auth_param basic credentialsttl 2 hours auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd authenticate_cache_garbage_interval 1 hour authenticate_ip_ttl 2 hours #acl all src 0.0.0.0/0.0.0.0 acl all src all acl manager proto cache_object acl localhost src 127.0.0.1 acl cacheadmin src 88.xxx.xxx.xxx 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 1863 # MSN messenger acl ncsa_users proxy_auth REQUIRED acl maxuser max_user_ip -s 2 acl CONNECT method CONNECT http_access allow manager cacheadmin http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access deny to_localhost http_access deny manager http_access allow ncsa_users http_access deny maxuser http_access deny all icp_access allow all http_port 8080 http_port 88.xxx.xxx.xxx:80 hierarchy_stoplist cgi-bin ? cache_mem 256MB maximum_object_size_in_memory 50 KB cache_replacement_policy heap LFUDA cache_dir aufs /var/spool/squid 40000 16 256 maximum_object_size 50 MB cache_swap_low 90 cache_swap_high 95 access_log /var/log/squid/access.log squid cache_log /var/log/squid/cache.log buffered_logs on refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 quick_abort_min 0 KB quick_abort_max 0 KB acl apache rep_header Server ^Apache broken_vary_encoding allow apache half_closed_clients off cache_mgr ***'***.com cachemgr_passwd ******** all visible_hostname P*****r ProxyServer log_icp_queries off dns_nameservers 208.67.222.222 208.67.220.220 hosts_file /etc/hosts memory_pools off forwarded_for off client_db off coredump_dir /var/spool/squid > ---------------------------------------- >> From: webster_jack@xxxxxxxxxxx >> To: squid3@xxxxxxxxxxxxx; squid-users@xxxxxxxxxxxxxxx >> Date: Sat, 13 Feb 2010 16:35:29 +0000 >> Subject: RE: Cache manager analysis >> >> >> Thanks. >> A few questions on this: >> (a) when you said this all src all is that meant to be acl src all? >> (b) Hint 2: if possible, define an ACL or the network ranges where you accept logins. Use it like so >> The logins are accepted form IP addresses that I never know, it is an external proxy server for geo location so not sure I can do this? logins will only ever by directed to the 88.xxx.xxx.xxx server though? >> (c) cache_mem 100 MB >> Bump this up as high as you can go without risking memory swapping. >> Objects served from RAM are 100x faster than objects not. >> Where can I view if memeory swapping is happening? >> (D) maximum_object_size 50 MB >> Bump this up too. Holding full ISO CDs and windows service packs can >> boost performance when one is used from the cache. 40GB of disk can >> store a few. >> If I increase this, will the server ever try to store streamed video? I had an efficiency problem with the original configuration that came with squid, which meant that streamed video was buffering constantly. Not sure what caused it but with the current config it does not do that. >> If I increase the cache_mem and max object size do I also need to increase this? >> maximum_object_size_in_memory 50 KB >> (E) >> cache_swap_low 90 >> cache_swap_high 95 >> access_log /var/log/squid/access.log squid >> cache_log /var/log/squid/cache.log >> buffered_logs on >> acl QUERY urlpath_regex cgi-bin \? >> cache deny QUERY >> >> Drop the QUERY bits above. It's more than halving the things your Squid can store. >> Remove the acl and the cache deny? >> At present, does this stop the cache from storing anything with a ?, ie dynamic pages? >> What if the same request is made for a dynamic page, will it retrive it from the cache (old page) rather then fetch the new dynamic content? >> >> current conf redone below: >> ---------------------------- >> auth_param basic realm Proxy server >> auth_param basic credentialsttl 2 hours >> auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd >> authenticate_cache_garbage_interval 1 hour >> authenticate_ip_ttl 2 hours >> #acl all src 0.0.0.0/0.0.0.0 >> acl src all >> acl manager proto cache_object >> acl localhost src 127.0.0.1 >> acl cacheadmin src 88.xxx.xxx.xxx >> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 >> acl SSL_ports port 443 >> acl Safe_ports port 80 # http >> acl Safe_ports port 21 # ftp >> acl Safe_ports port 443 # https >> acl Safe_ports port 70 # gopher >> acl Safe_ports port 210 # wais >> acl Safe_ports port 1025-65535 # unregistered ports >> acl Safe_ports port 280 # http-mgmt >> acl Safe_ports port 488 # gss-http >> acl Safe_ports port 591 # filemaker >> acl Safe_ports port 777 # multiling http >> acl Safe_ports port 1863 # MSN messenger >> acl ncsa_users proxy_auth REQUIRED >> acl maxuser max_user_ip -s 2 >> acl CONNECT method CONNECT >> http_access allow manager localhost >> http_access allow manager cacheadmin >> http_access deny !Safe_ports >> http_access deny CONNECT !SSL_ports >> http_access deny to_localhost >> http_access deny manager >> http_access allow ncsa_users >> http_access deny maxuser >> #http_access allow localhost >> http_access deny all >> icp_access allow all >> http_port 8080 >> http_port 88.xxx.xxx.xxx:80 >> hierarchy_stoplist cgi-bin ? >> cache_mem 100 MB >> maximum_object_size_in_memory 50 KB >> cache_replacement_policy heap LFUDA >> cache_dir aufs /var/spool/squid 40000 16 256 >> maximum_object_size 50 MB >> cache_swap_low 90 >> cache_swap_high 95 >> access_log /var/log/squid/access.log squid >> cache_log /var/log/squid/cache.log >> buffered_logs on >> #acl QUERY urlpath_regex cgi-bin \? >> #cache deny QUERY >> refresh_pattern ^ftp: 1440 20% 10080 >> refresh_pattern ^gopher: 1440 0% 1440 >> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >> refresh_pattern . 0 20% 4320 >> quick_abort_min 0 KB >> quick_abort_max 0 KB >> acl apache rep_header Server ^Apache >> broken_vary_encoding allow apache >> half_closed_clients off >> cache_mgr aaa@xxxxxxx >> cachemgr_passwd aaa all >> visible_hostname ProxyServer >> log_icp_queries off >> dns_nameservers 208.67.222.222 208.67.220.220 >> hosts_file /etc/hosts >> memory_pools off >> forwarded_for off >> client_db off >> coredump_dir /var/spool/squid >> >> ---------------------------------------- >>> Date: Sat, 13 Feb 2010 18:03:00 +1300 >>> From: squid3@xxxxxxxxxxxxx >>> To: squid-users@xxxxxxxxxxxxxxx >>> Subject: Re: Cache manager analysis >>> >>> J. Webster wrote: >>>> What is the best place to start with in cache analysis? >>>> Would it be cache size, memory object size, IO, etc.? >>>> I'm looking to optimise the settings for my squid server. >>> >>> Step 0) migrate to the latest Squid 2.7 or 3.1 or if possible 2.HEAD >>> (that one is only nominally beta, it's very stable in reality) >>> >>> 1) Start by defining 'optimize' ... are you going to prioritize... >>> Faster service? >>> More bandwidth saving? >>> More client connections? >>> >>> 2a) For faster service, look at DNS delays, disk IO delays, maximizing >>> cacheable objects (dynamic objects etc). >>> >>> 2b) For pure bandwidth savings start with a look at object cacheablity. >>> Check dynamics are being cached, ranges are being fetched in full, etc >>> >>> 3) Then profile all the objects stored over a reasonably long period, >>> looking at size. compare with the age of objects being discarded. >>> >>> 3a) tune the storage limits to prioritize the storage locations. giving >>> priority to RAM, then COSS, then AUFS/diskd. >>> >>> 3b) set the storage limits as high as possible to maximize amount of >>> data stored. anywhere. >>> >>> 4) take a good long look at your access controls and in particular the >>> types speedy/fast/slow. You may get some speed benefits from fixing up >>> the ordering a bit. regex are killers, remote lookups (helpers, or DNS) >>> are second worst. >>> (some performance hints below) >>> >>> 5) repeat from (2b) as often as possible. concentrate traffic which >>> seems to logically be storeable but gets a TCP_MISS anyway. >>> >>> Objects served from cache lead to faster service ties for those objects, >>> so the speed vs bandwidth are inter-related somewhat. But there is a >>> tipping point somewhere where tuning one starts to impact the other. >>> >>> >>>> >>>> Server: about 220GB available for the cache, I'm only using 40000 MB at present as in the config below. >>>> system D2812-A2 >>>> /0 bus D2812-A2 >>>> /0/0 memory 110KiB BIOS >>>> /0/4 processor Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz >>>> /0/4/5 memory 64KiB L1 cache >>>> /0/4/6 memory 3MiB L2 cache >>>> /0/4/0.1 processor Logical CPU >>>> /0/4/0.2 processor Logical CPU >>>> /0/7 memory 3MiB L3 cache >>>> /0/2a memory 1GiB System Memory >>>> /0/2a/0 memory 1GiB DIMM DDR2 Synchronous 667 MHz (1.5 ns) >>>> /0/2a/1 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>> /0/2a/2 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>> /0/2a/3 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>> /0/1 processor >>>> /0/1/0.1 processor Logical CPU >>>> /0/1/0.2 processor Logical CPU >>>> >>>> >>>> Current squid.conf: >>>> --------------------- >>>> auth_param basic realm Proxy server >>>> auth_param basic credentialsttl 2 hours >>>> auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd >>>> authenticate_cache_garbage_interval 1 hour >>>> authenticate_ip_ttl 2 hours >>>> acl all src 0.0.0.0/0.0.0.0 >>> >>> all src all >>> >>>> acl manager proto cache_object >>>> acl localhost src 127.0.0.1/255.255.255.255 >>> >>> acl localhost src 127.0.0.1 >>> >>>> acl cacheadmin src 88.xxx.xxx.xxx >>>> acl to_localhost dst 127.0.0.0/8 >>> >>> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 >>> >>>> acl SSL_ports port 443 >>>> acl Safe_ports port 80 # http >>>> acl Safe_ports port 21 # ftp >>>> acl Safe_ports port 443 # https >>>> acl Safe_ports port 70 # gopher >>>> acl Safe_ports port 210 # wais >>>> acl Safe_ports port 1025-65535 # unregistered ports >>>> acl Safe_ports port 280 # http-mgmt >>>> acl Safe_ports port 488 # gss-http >>>> acl Safe_ports port 591 # filemaker >>>> acl Safe_ports port 777 # multiling http >>>> acl Safe_ports port 1863 # MSN messenger >>>> acl ncsa_users proxy_auth REQUIRED >>>> acl maxuser max_user_ip -s 2 >>>> acl CONNECT method CONNECT >>>> http_access allow manager localhost >>>> http_access allow manager cacheadmin >>> >>> Hint: add the localhost IP to the cacheadmin ACL and drop one full set >>> of "allow manager localhost" tests. >>> >>>> http_access deny manager >>>> http_access allow ncsa_users >>> >>> Hint: drop the authentication down ... >>> >>>> http_access deny !Safe_ports >>>> http_access deny CONNECT !SSL_ports >>>> http_access deny to_localhost >>> >>> ... to here. All the attacks against your proxy for bad ports and >>> sources will be dropped quickly by the security blanket settings. Load >>> on your auth server will reduce and may speed up it's response time. >>> >>> Hint 2: if possible, define an ACL or the network ranges where you >>> accept logins. Use it like so: >>> >>> http_access allow localnet ncsa_users >>> >>> ... once again that speeds up the rejections, and helps by reducing >>> the number of times the slow auth lookup needs checking. >>> >>>> http_access deny maxuser >>>> http_access allow localhost >>> >>> If localhost really is allowed to do anything, move it up above the >>> "to_localhost" one. >>> Otherwise drop this completely, having the correct auth login details >>> will permit links from localhost just as easily as from anywhere else. >>> >>>> http_access deny all >>>> icp_access allow all >>> >>> Define the networks where peer siblings are trusted. Allwo them and deny >>> everything else. >>> That will reduce a fair bit of load on your Squid trying to service >>> random ICP requests from the general Internet. >>> >>>> http_port 8080 >>>> http_port 88.xxx.xxx.xxx:80 >>>> hierarchy_stoplist cgi-bin ? >>>> cache_mem 100 MB >>> >>> Bump this up as high as you can go without risking memory swapping. >>> Objects served from RAM are 100x faster than objects not. >>> >>>> maximum_object_size_in_memory 50 KB >>>> cache_replacement_policy heap LFUDA >>>> cache_dir aufs /var/spool/squid 40000 16 256 >>> >>> If you pick 2.x squid to upgrade to, add a COSS directory as well. >>> See the recent threads on optimizing COSS for how to tune that. >>> >>>> maximum_object_size 50 MB >>> >>> Bump this up too. Holding full ISO CDs and windows service packs can >>> boost performance when one is used from the cache. 40GB of disk can >>> store a few. >>> >>>> cache_swap_low 90 >>>> cache_swap_high 95 >>>> access_log /var/log/squid/access.log squid >>>> cache_log /var/log/squid/cache.log >>>> buffered_logs on >>>> acl QUERY urlpath_regex cgi-bin \? >>>> cache deny QUERY >>> >>> Drop the QUERY bits above. It's more than halving the things your Squid >>> can store. >>> >>>> refresh_pattern ^ftp: 1440 20% 10080 >>>> refresh_pattern ^gopher: 1440 0% 1440 >>> >>> Add right here: >>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >>> >>>> refresh_pattern . 0 20% 4320 >>>> quick_abort_min 0 KB >>>> quick_abort_max 0 KB >>>> acl apache rep_header Server ^Apache >>>> broken_vary_encoding allow apache >>>> half_closed_clients off >>>> cache_mgr aaa@xxxxxxx >>>> cachemgr_passwd aaa all >>>> visible_hostname ProxyServer >>>> log_icp_queries off >>>> dns_nameservers 208.67.222.222 208.67.220.220 >>>> hosts_file /etc/hosts >>>> memory_pools off >>> >>> Might cause efficiency problems if the underlying malloc is not >>> optimized. but oh well, up to you. >>> >>>> forwarded_for off >>>> client_db off >>>> coredump_dir /var/spool/squid >>>> >>> >>> >>> Amos >>> -- >>> Please be using >>> Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23 >>> Current Beta Squid 3.1.0.16 >> >> _________________________________________________________________ >> Send us your Hotmail stories and be featured in our newsletter >> http://clk.atdmt.com/UKM/go/195013117/direct/01/ > > _________________________________________________________________ > Send us your Hotmail stories and be featured in our newsletter > http://clk.atdmt.com/UKM/go/195013117/direct/01/ _________________________________________________________________ We want to hear all your funny, exciting and crazy Hotmail stories. Tell us now http://clk.atdmt.com/UKM/go/195013117/direct/01/
