That indicates that when the cache dir gets more than 95% full it needs
to clear at least 2GB of objects (2% of 115,000MB).
Depending on your disks and avg object sizes this may be a large part of
the load.
As for the regex acls since like I have said these specified by the
videocache python squid plugin redirector setup I can try turning all of
these off and see if the problem persists or no.
And these regex acls are used only with below access lines
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
A lot of those can be reduced to dstdomain or at least dstdomain
followed by a regex.
ie:
acl A dstdomain .googlevideo.com
acl R urlpath_regex -i /videoplayback
http_access allow/deny A R
Using dstdomain (very fast test) at the start of an access line to
determine if the slower regex later on in the line is even needed can
produce a dramatic speed boost in response times.
Where you are testing for file types the rep_mime_type on reply can
check the mime faster than a regex can test the entire URL.
Also matching all of that file type by mime from a whole dstdomain can
be more effective and faster than a specific regex testing for a range
of specific sub-domains and file endings.
Also you do not indicate which if the *_access lines these are being
tested in. It's a very good idea to only to them in one of either
http_access or http_reply_access, not both.
Ragheb Rustom
-----Original Message-----
From: Amos Jeffries [mailto:squid3@xxxxxxxxxxxxx]
Sent: Saturday, May 30, 2009 11:16 AM
To: Ragheb Rustom
Cc: squid-users@xxxxxxxxxxxxxxx; crobertson@xxxxxxx
Subject: Re: squid2.7 Stable 6 passes in intervals where cpu
usage hits 100%
Ragheb Rustom wrote:
Hi Chris,
Thank you for the reply. I will use squid -k debug to check what is going
on.
As for the additional info sorry for not clearing the squid setup more
anyway this is squid configuration values you have guessed as taken from
my
squid.conf
Wild guesses based on the information given:
*) You are using UFS for your cache_dir instead of AUFS.
cache_dir aufs /cache1 115000 16 256
cache_dir aufs /cache2 115000 16 256
I am using AUFS not UFS.
*) Your cache is pretty large and your cache_swap_high and
cache_swap_low are still at the defaults
cache_swap_low 93
cache_swap_high 95
Do u think these values are reasonable enough regarding the above values
of
my Cacche dir or should they be changed.
That indicates that when the cache dir gets more than 95% full it needs
to clear at least 2GB of objects (2% of 115,000MB).
Depending on your disks and avg object sizes this may be a large part of
the load.
*) You are using large lists of regex ACLs
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback
\.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback
\.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback
\.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]
?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][
0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]
?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][
0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i
proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i
[a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i
va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i
\.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i
media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i
\.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|r
am)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain v.mccont.com dl.redtube.com
.cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com
http:\/\/www\.youtube\.com
acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh default\.ida?XXXXXXX
omg.pif
acl music url_regex -i .wav
acl video url_regex -i .avi .mpe
acl cddown url_regex -i .iso .raw
These are all the acl regex entries I use in my squid.conf file mainly
most
of them are those of videocache.
A lot of those can be reduced to dstdomain or at least dstdomain
followed by a regex.
ie:
acl A dstdomain .googlevideo.com
acl R urlpath_regex -i /videoplayback
http_access allow/deny A R
Using dstdomain (very fast test) at the start of an access line to
determine if the slower regex later on in the line is even needed can
produce a dramatic speed boost in response times.
Where you are testing for file types the rep_mime_type on reply can
check the mime faster than a regex can test the entire URL.
Also matching all of that file type by mime from a whole dstdomain can
be more effective and faster than a specific regex testing for a range
of specific sub-domains and file endings.
Also you do not indicate which if the *_access lines these are being
tested in. It's a very good idea to only to them in one of either
http_access or http_reply_access, not both.
I will use the squid -k debug and try and get more info of what is going
on
during the high cpu usage period and will share whatever I find on
squid-users mail list.
Thanks Chris for your support.
Ragheb
-----Original Message-----
From: crobertson@xxxxxxx [mailto:crobertson@xxxxxxx]
Sent: Friday, May 29, 2009 10:18 PM
To: squid-users@xxxxxxxxxxxxxxx
Subject: Re: squid2.7 Stable 6 passes in intervals where cpu
usage hits 100%
Ragheb Rustom wrote:
Dear All,
I have been noticing for some time now that my squid server passes in
nearly
regular intervals during the day where the squid process CPU usage hits
100%
for around 20 - 30 seconds or so during which squid will not serve
requests.
After these 20 -30sec the squid CPU usage drops to around 5 - 13% and
website navigation is normal again. This happens like I said before at
nearly regular intervals during each day several times and it happens
only
for around 20 -30sec. I have been observing the cache.log file but there
is
nothing apparent there. Is there a way that I know what squid is doing
when
this problem happens to try and resolve it as trying to solve it like it
is
now with no info about what is going on is somewhat vague.
squid -k debug
I think this
problem happened after I have changed memory and disk replacement
policies
from lru to heap but I am not so sure that this is the source of the
problem. I am using squid2.7-Stable6 on a fedora9 server which is a dual
xeon quad 2.8Ghz CPU with 16Gb of Ram installed and 3 SAS hard drives.
One
more thing this server is also running videocache 1.9.1 but it has been
running videocache also before the problem started with no problems
whatsoever. I appreciate any help you can offer me.
Wild guesses based on the information given:
*) You are using UFS for your cache_dir instead of AUFS.
*) Your cache is pretty large and your cache_swap_high and
cache_swap_low are still at the defaults
*) You are using large lists of regex ACLs
Sincerely,
Ragheb Rustom
Smartelecom S.A.R.L
Sin el fil - Mar Elias Street
Absi Center - Ground Floor
Beirut, Lebanon
Email: ragheb@xxxxxxxxxxxxxxx
Chris
