Juz,
The mount options rw,noatime reduce I/O a little for ext4 so they are recommended for /squid.
Since the system has 4 GB memory it is recommendable to increase
cache_mem from 32 MB to 512 MB and to change
maximum_object_size_in_memory from 20 KB to 128 KB.
Both options help to cache more in-memory instead of on-disk and hence reduce disk reads.
But only increase the parameters if the system has enough free memory to give to Squid.
Note that 512 MB memory cache 'translates' into roughly 1.4 GB total memory requirement for Squid.
And last but not least, since the disk is a virtual disk, it is recommendable
to see if the virtual disk can be allocated on a device with more I/O capacity.
Marcus
On 07/25/2014 05:52 AM, RYAN Justin wrote:
Sorry Marcus, new to this forum support.
You mention cache_mem is small, excuse me noobness can you explain the impact.
The Memory allocation to the VM is 4GB, and it has at present 4 VCPU (doesn't look like it being stressed at all).
Version = Squid Cache: Version 3.2.5
Disk structure is as follows
20GB VDMK = System
40GB VMDK = SQUID only
#
# /etc/fstab
# Created by anaconda on Mon Apr 23 16:24:28 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_008-lv_root / ext4 defaults 1 1
UUID=c13ba480-17e3-4df3-b6d3-9a2eb9cea766 /boot ext4 defaults 1 2
# UUID=08301dc8-4e84-4cd9-a402-f4e71a461098 /squid ext4 defaults 1 2
/dev/mapper/vg_008-lv_swap swap swap defaults 0 0
/dev/sdb /squid ext4 defaults 1 2
-----Original Message-----
From: Marcus Kool [mailto:marcus.kool@xxxxxxxxxxxxxxx]
Sent: 25 July 2014 00:37
To: RYAN Justin
Subject: Re: FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged
Juz,
The systems seems to have a very small config.
32 MB for cache_mem is very small indeed Do you have room/RAM to extend the in-memory cache of Squid?
From the data that you posted it is not clear if /squid shares its disk with /.
What version of Squid do you have (output of squid -v) ?
What file system type and mount options are used for /squid ?
You did not reply to the squid list.
I suggest to include the squid list in the CC: and replace the cachemgr_passwd to XXX in the post.
Marcus
On 07/24/2014 10:39 AM, RYAN Justin wrote:
Sorry Marcus, was a little light on background. Storage on 2
partitions
[root@ ]# df -k Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2057264 0 2057264 0% /dev
tmpfs 2066040 0 2066040 0% /dev/shm
tmpfs 2066040 504 2065536 1% /run
/dev/mapper/vg_008-lv_root 16062384 3864120 11382344 26% /
tmpfs 2066040 0 2066040 0% /sys/fs/cgroup
tmpfs 2066040 0 2066040 0% /media
/dev/sdb 41284928 14322924 24864852 37% /squid
/dev/sda2 495844 65891 404353 15% /boot
Below is the config
http_port 3128
dns_nameservers 8.8.8.8
icp_port 0
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
append_domain .phoenix.loc
cache_mgr it@xxxxxxxxx
cachemgr_passwd ***** all
buffered_logs on
coredump_dir /squid/cache
cache_access_log /squid/logs/access.log
cache_log /squid/logs/cache.log
logfile_rotate 60
cache_dir aufs /squid/cache 4096 16 256 cache_mem 32 MB
maximum_object_size 64 MB
maximum_object_size_in_memory 20 KB
cache_effective_user squid max_filedesc 4096
# acl all src all
# acl manager proto cache_object
acl localhost src 127.0.0.1
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 # https
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
#acl SSL method CONNECT
acl CONNECT method CONNECT
acl webserver src 192.168.100.0/24
http_access allow manager webserver
http_access allow manager localhost
http_access deny manager
http_access deny CONNECT !SSL_ports
http_access deny !Safe_ports
http_access allow localhost
# -------------------------------------------------------
auth_param ntlm program /usr/bin/ntlm_auth
--helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 30 startup=30 # auth_param ntlm
use_ntlm_negotiate on auth_param ntlm keep_alive off
auth_param basic program /usr/bin/ntlm_auth
--helper-protocol=squid-2.5-basic auth_param basic children 10
startup=10 auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
external_acl_type ADS children-max=30 children-startup=30 %LOGIN
/usr/lib/squid/ext_wbinfo_group_acl
acl block_all dstdomain "/squid/rules/block-all acl malware dstdomain
"/squid/rules/malware.jr"
acl download_all urlpath_regex \.mp1$ \.mp2$ \.mp3$ \.mpa$ \.mpeg$
\.asf$ \.mpg$ \.vob$ \.scr$ \.avi$ \.mtm$ \.orc$ \.rnx$ \.sng$ \$
\.gzip$ \.wma$ http_access deny block_all http_access deny malware
http_access deny download_all
acl retail_static src "/squid/rules/retail-clients"
acl word-control url_regex -i "/squid/rules/word-control"
acl site-control dstdomain "/squid/rules/site-control"
acl download urlpath_regex \.exe$ \.zip$ http_access deny
retail_static word-control http_access deny retail_static site-control
http_access deny retail_static download
acl blackberry-access src 192.168.100.50 http_access allow
blackberry-access
acl navman url_regex -i "/squid/rules/navman"
http_access allow navman
acl myp-i-n dstdomain "/squid/rules/home-sites"
acl domain-home dst "/squid/rules/home-dest"
acl home-regex url_regex -i "/squid/rules/home-regex"
http_access allow myp-i-n
http_access allow domain-home
http_access allow home-regex
acl retail_ras external ADS PA-Access-Retail acl retail-branch
dstdomain "/squid/rules/retail-sites"
http_access allow retail_ras retail-branch
acl access-social external ADS PA-Access-Social
sg_UK_TM7_Proxy_Social_Media acl social-sites dstdomain "/squid/rules/social-sites"
http_access allow access-social social-sites http_access deny
social-sites
acl access-streaming external ADS PA-Access-Streaming
sg_UK_TM7_Proxy_Streaming_Media acl streaming-sites dstdomain "/squid/rules/streaming-sites"
http_access allow access-streaming streaming-sites http_access deny
streaming-sites
acl access-york external ADS PA-Access-York acl york-sites dstdomain
"/squid/rules/york-sites"
http_access allow access-york york-sites
acl access-training external ADS PA-Access-Training acl training-sites
dstdomain "/squid/rules/training-sites"
http_access allow access-training training-sites
acl access-finance external ADS PA-Access-Finance acl finance-sites
dstdomain "/squid/rules/finance-sites"
http_access allow access-finance finance-sites
acl access-others external ADS PA-Access-Others acl other-sites
dstdomain "/squid/rules/authorised"
http_access allow access-others other-sites
acl internet_access external ADS PA-Access-Complete
sg_UK_TM7_Proxy_Complete_Access http_access allow internet_access
http_access deny all
http_access deny all
-----Original Message-----
From: Marcus Kool [mailto:marcus.kool@xxxxxxxxxxxxxxx]
Sent: 24 July 2014 14:02
To: RYAN Justin; 'squid-users@xxxxxxxxxxxxxxx'
Subject: Re: FW: Problem with server IO resource, need
to reduce logging level by excluding specific sites from being logged
Juz,
It helps if you describe the system in more detail.
What is the configuration of Squid (squid.conf without the comments) and how are the data store file systems spread over the disks ?
For immediate results, you can reduce the disk cache or even temporarily disable the disk cache.
Marcus
On 07/24/2014 09:36 AM, RYAN Justin wrote:
Hi,
Probably been answered before but my problem is this.
We have a esx server out in our providers pop. The server has been in situ for quite sometime. We have noticed that the datastore writes latency has increased to on average 45ms. We know this is bad but have a problem both budgetry and timescale and require a quick solution.
We propose to reduce IO load by excluding specific traffic from getting logged, but are now looking on how to achieve this.
Understand a change to squid.conf is required plus the use of a access list. Problem being we have inherited this build and there is no real documentation.
Holding our hands up, we are not experts and could really do with some community help.
Thanks in advance.
Juz
PRIVACY & CONFIDENTIALITY
This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation.
Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any).
This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services.
PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370
PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington
Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No.
741706443
PRIVACY & CONFIDENTIALITY
This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation.
Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any).
This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services.
PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370
PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington
Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No.
741706443
PRIVACY & CONFIDENTIALITY
This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation.
Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any).
This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services.
PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370
PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234
Rivington Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ
VAT No. 741706443
