On 27/03/2012 12:25, Christian Loth wrote:
Hello,
first of all, thank you for your recommendations.
On Monday 26 March 2012 16:34:21 Marcus Kool wrote:
Youtube may be hogging your pipe but it is better to know than to guess.
Of course, before we decided for the proxy setup we investigated bandwidth
usage. HTTP Traffic was about 60-70% of our traffic, and a good chunk of that was
youtube. That's why we decided, that we would try a squid setup with youtube
caching. However it took a while before we found a solution for caching
youtube, as 3.1 hasn't implemented the necessary features yet.
The access.log shows content sizes so with a simple awk script it should
be easy to find out.
I have also seen many sites where advertisements and trackers consume 15%
bandwidth. This may vary. So blocking ads and trackers is a thing to
consider.
Thanks for this insight! This would of course be a welcome saving of bandwidth
in my personal opinion. I'm just not sure if we're allowed to do this, as the
patron of the proxy is a public-law institution and as such bound to anti-
censorship laws. Need to check with a Legalese translator.
Do not expect too much from web caching. More and more websites
intentionally make their sites not cacheable. Look at the percentage of
TCP_MISS in access.log or use a second awk script to find out more about
cacheability.
Every bit counts. Before we apply for an increase of (expensive) uplink
bandwidth we want to play every trick we have up our sleeve. At the moment our
cache is still cold, because for getting the proxy running again I had to
completely wipe the cache. At the moment we have a hit:miss ratio of about
1:5. For youtube caching we have a saved bandwidth around 100 GB for the 27th
of march (one video in particular had a size of 768 MB and was watched 19
times). Online lectures are currently en vogue.
I recommend going for a newer Squid: 3.1.19 is stable and fixes issues that
3.1.10 has.
Will do so.
On Linux, aufs has a better performance than diskd
Thanks again for this tip!
Additional memory for storing objects is 2048 MB:
cache_mem 2048 MB
Seems right. But you also need virtual memory for Squid being able to
fork processes without issues. Do have have 8 GB swap ?
Yes. 10 GB actually.
But read the FAQ about memory usage and a large disk cache:
http://wiki.squid-cache.org/SquidFaq/SquidMemory
Squid uses an additional 512*14 MB = 7.1 GB for the index of the disk
cache. I suggest to downsize to 1 GB in-memory index which implies to
use only 73 GB disk cache.
Ah okay, here's one of my initial mistakes. I used only 10 MB for my
calculation but of course we use a 64bit squid. Out of curiosity and because I
want to learn: the reasoning for shrinking the disk cache from 512 GB to 73 GB
is because a big cache as we have it at the moment only leads to lots of stale
objects in cache which additionally burden the CPU and RAM because of in-
memory metadata?
mostly because for a squid cache (not nginx) of 512GB will consume a lot
of memory on index while you will prefer to serve other stuff from
memory and because of the stale objects.
if you are using nginx for youtube caching remember to eliminate the
caching on youtube videos.
if you wont what will happen is that nginx will create new headers for
the cached objects and then squid will cache them but will never use
them again.
(if you didnt changed the basic not cache for objects contains "?" and
"cgi-bin" you are safe).
by the way if you have some time to analyze the proxy logs you can
manage to find some other sites you can use nginx to serve some stale files.
i have used it to cache also windows updates and some other video sites
but the pattern was similar so i had a huge list i could serv most of
the stale content from nginx what was leaving a lot of ram for squid index.
p.s.
forgot to say that you always must not get into a point that your proxy
ram is at the limit or else swapiness will happen and will slow the
server down.
Regards,
Eliezer
Best regards,
- Christian Loth
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il
