Cheers Marcus, I did see via googling a rule of thumb quote " cache_mem = total physical memory / 3" - ref http://forums.justlinux.com/showthread.php?126396-Squid-cache-tuning there is a more complex formula quoted too. Money and access constraints negate the move to faster storage :) I will look into your recommendations. The question of removing noise from being logged still exists - would be a nice to have option -----Original Message----- From: Marcus Kool [mailto:marcus.kool@xxxxxxxxxxxxxxx] Sent: 25 July 2014 12:11 To: RYAN Justin Cc: 'squid-users@xxxxxxxxxxxxxxx' Subject: Re: FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged Juz, The mount options rw,noatime reduce I/O a little for ext4 so they are recommended for /squid. Since the system has 4 GB memory it is recommendable to increase cache_mem from 32 MB to 512 MB and to change maximum_object_size_in_memory from 20 KB to 128 KB. Both options help to cache more in-memory instead of on-disk and hence reduce disk reads. But only increase the parameters if the system has enough free memory to give to Squid. Note that 512 MB memory cache 'translates' into roughly 1.4 GB total memory requirement for Squid. And last but not least, since the disk is a virtual disk, it is recommendable to see if the virtual disk can be allocated on a device with more I/O capacity. Marcus On 07/25/2014 05:52 AM, RYAN Justin wrote: > Sorry Marcus, new to this forum support. > > You mention cache_mem is small, excuse me noobness can you explain the impact. > The Memory allocation to the VM is 4GB, and it has at present 4 VCPU (doesn't look like it being stressed at all). > > Version = Squid Cache: Version 3.2.5 > > Disk structure is as follows > > 20GB VDMK = System > 40GB VMDK = SQUID only > > # > # /etc/fstab > # Created by anaconda on Mon Apr 23 16:24:28 2012 # # Accessible > filesystems, by reference, are maintained under '/dev/disk' > # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more > info # > /dev/mapper/vg_008-lv_root / ext4 defaults 1 1 > UUID=c13ba480-17e3-4df3-b6d3-9a2eb9cea766 /boot ext4 defaults 1 2 > # UUID=08301dc8-4e84-4cd9-a402-f4e71a461098 /squid ext4 defaults 1 2 > /dev/mapper/vg_008-lv_swap swap swap defaults 0 0 > /dev/sdb /squid ext4 defaults 1 2 > > -----Original Message----- > From: Marcus Kool [mailto:marcus.kool@xxxxxxxxxxxxxxx] > Sent: 25 July 2014 00:37 > To: RYAN Justin > Subject: Re: FW: Problem with server IO resource, need > to reduce logging level by excluding specific sites from being logged > > Juz, > > The systems seems to have a very small config. > 32 MB for cache_mem is very small indeed Do you have room/RAM to extend the in-memory cache of Squid? > > From the data that you posted it is not clear if /squid shares its disk with /. > > What version of Squid do you have (output of squid -v) ? > > What file system type and mount options are used for /squid ? > > You did not reply to the squid list. > I suggest to include the squid list in the CC: and replace the cachemgr_passwd to XXX in the post. > > Marcus > > > > On 07/24/2014 10:39 AM, RYAN Justin wrote: >> Sorry Marcus, was a little light on background. Storage on 2 >> partitions >> >> [root@ ]# df -k Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2057264 0 2057264 0% /dev >> tmpfs 2066040 0 2066040 0% /dev/shm >> tmpfs 2066040 504 2065536 1% /run >> /dev/mapper/vg_008-lv_root 16062384 3864120 11382344 26% / >> tmpfs 2066040 0 2066040 0% /sys/fs/cgroup >> tmpfs 2066040 0 2066040 0% /media >> /dev/sdb 41284928 14322924 24864852 37% /squid >> /dev/sda2 495844 65891 404353 15% /boot >> >> Below is the config >> >> http_port 3128 >> dns_nameservers 8.8.8.8 >> icp_port 0 >> acl QUERY urlpath_regex cgi-bin \? >> no_cache deny QUERY >> append_domain .phoenix.loc >> >> cache_mgr it@xxxxxxxxx >> cachemgr_passwd ***** all >> >> buffered_logs on >> coredump_dir /squid/cache >> >> cache_access_log /squid/logs/access.log >> >> cache_log /squid/logs/cache.log >> logfile_rotate 60 >> >> cache_dir aufs /squid/cache 4096 16 256 cache_mem 32 MB >> maximum_object_size 64 MB maximum_object_size_in_memory 20 KB >> cache_effective_user squid max_filedesc 4096 >> >> >> # acl all src all >> # acl manager proto cache_object >> acl localhost src 127.0.0.1 >> acl to_localhost dst 127.0.0.0/8 >> acl SSL_ports port 443 # https >> acl Safe_ports port 80 # http >> acl Safe_ports port 21 # ftp >> acl Safe_ports port 443 # https >> acl Safe_ports port 70 # gopher >> acl Safe_ports port 210 # wais >> acl Safe_ports port 1025-65535 # unregistered ports >> acl Safe_ports port 280 # http-mgmt >> acl Safe_ports port 488 # gss-http >> acl Safe_ports port 591 # filemaker >> acl Safe_ports port 777 # multiling http >> #acl SSL method CONNECT >> acl CONNECT method CONNECT >> >> acl webserver src 192.168.100.0/24 >> http_access allow manager webserver >> >> http_access allow manager localhost >> http_access deny manager >> http_access deny CONNECT !SSL_ports >> http_access deny !Safe_ports >> http_access allow localhost >> >> # ------------------------------------------------------- >> auth_param ntlm program /usr/bin/ntlm_auth >> --helper-protocol=squid-2.5-ntlmssp >> auth_param ntlm children 30 startup=30 # auth_param ntlm >> use_ntlm_negotiate on auth_param ntlm keep_alive off >> >> auth_param basic program /usr/bin/ntlm_auth >> --helper-protocol=squid-2.5-basic auth_param basic children 10 >> startup=10 auth_param basic realm Squid proxy-caching web server >> auth_param basic credentialsttl 2 hours >> >> >> external_acl_type ADS children-max=30 children-startup=30 %LOGIN >> /usr/lib/squid/ext_wbinfo_group_acl >> >> >> >> acl block_all dstdomain "/squid/rules/block-all acl malware dstdomain >> "/squid/rules/malware.jr" >> acl download_all urlpath_regex \.mp1$ \.mp2$ \.mp3$ \.mpa$ \.mpeg$ >> \.asf$ \.mpg$ \.vob$ \.scr$ \.avi$ \.mtm$ \.orc$ \.rnx$ \.sng$ \$ >> \.gzip$ \.wma$ http_access deny block_all http_access deny malware >> http_access deny download_all >> >> >> acl retail_static src "/squid/rules/retail-clients" >> acl word-control url_regex -i "/squid/rules/word-control" >> acl site-control dstdomain "/squid/rules/site-control" >> acl download urlpath_regex \.exe$ \.zip$ http_access deny >> retail_static word-control http_access deny retail_static >> site-control http_access deny retail_static download >> >> >> >> acl blackberry-access src 192.168.100.50 http_access allow >> blackberry-access >> >> >> acl navman url_regex -i "/squid/rules/navman" >> http_access allow navman >> >> acl myp-i-n dstdomain "/squid/rules/home-sites" >> acl domain-home dst "/squid/rules/home-dest" >> acl home-regex url_regex -i "/squid/rules/home-regex" >> http_access allow myp-i-n >> http_access allow domain-home >> http_access allow home-regex >> >> >> acl retail_ras external ADS PA-Access-Retail acl retail-branch >> dstdomain "/squid/rules/retail-sites" >> http_access allow retail_ras retail-branch >> >> >> acl access-social external ADS PA-Access-Social >> sg_UK_TM7_Proxy_Social_Media acl social-sites dstdomain "/squid/rules/social-sites" >> http_access allow access-social social-sites http_access deny >> social-sites >> >> >> acl access-streaming external ADS PA-Access-Streaming >> sg_UK_TM7_Proxy_Streaming_Media acl streaming-sites dstdomain "/squid/rules/streaming-sites" >> http_access allow access-streaming streaming-sites http_access deny >> streaming-sites >> >> acl access-york external ADS PA-Access-York acl york-sites dstdomain >> "/squid/rules/york-sites" >> http_access allow access-york york-sites >> >> >> acl access-training external ADS PA-Access-Training acl >> training-sites dstdomain "/squid/rules/training-sites" >> http_access allow access-training training-sites >> >> acl access-finance external ADS PA-Access-Finance acl finance-sites >> dstdomain "/squid/rules/finance-sites" >> http_access allow access-finance finance-sites >> >> acl access-others external ADS PA-Access-Others acl other-sites >> dstdomain "/squid/rules/authorised" >> http_access allow access-others other-sites >> >> >> acl internet_access external ADS PA-Access-Complete >> sg_UK_TM7_Proxy_Complete_Access http_access allow internet_access >> http_access deny all >> >> http_access deny all >> >> >> >> -----Original Message----- >> From: Marcus Kool [mailto:marcus.kool@xxxxxxxxxxxxxxx] >> Sent: 24 July 2014 14:02 >> To: RYAN Justin; 'squid-users@xxxxxxxxxxxxxxx' >> Subject: Re: FW: Problem with server IO resource, need >> to reduce logging level by excluding specific sites from being logged >> >> Juz, >> >> It helps if you describe the system in more detail. >> What is the configuration of Squid (squid.conf without the comments) and how are the data store file systems spread over the disks ? >> >> For immediate results, you can reduce the disk cache or even temporarily disable the disk cache. >> >> Marcus >> >> On 07/24/2014 09:36 AM, RYAN Justin wrote: >>> >>> Hi, >>> Probably been answered before but my problem is this. >>> We have a esx server out in our providers pop. The server has been in situ for quite sometime. We have noticed that the datastore writes latency has increased to on average 45ms. We know this is bad but have a problem both budgetry and timescale and require a quick solution. >>> >>> We propose to reduce IO load by excluding specific traffic from getting logged, but are now looking on how to achieve this. >>> Understand a change to squid.conf is required plus the use of a access list. Problem being we have inherited this build and there is no real documentation. >>> >>> Holding our hands up, we are not experts and could really do with some community help. >>> Thanks in advance. >>> Juz >>> >>> >>> PRIVACY & CONFIDENTIALITY >>> >>> This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation. >>> >>> Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any). >>> >>> This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services. >>> >>> PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370 >>> PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington >>> Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No. >>> 741706443 >>> >>> >> >> PRIVACY & CONFIDENTIALITY >> >> This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation. >> >> Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any). >> >> This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services. >> >> PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370 >> PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington >> Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No. >> 741706443 >> >> > > PRIVACY & CONFIDENTIALITY > > This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation. > > Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any). > > This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services. > > PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370 > PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington > Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No. > 741706443 > > PRIVACY & CONFIDENTIALITY This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation. Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any). This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services. PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370 PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No. 741706443
