David Tosoff wrote:
All,
I'm new to Squid and I have been given the task of optimizing the delivery of photos from our website. We have 1 main active image server which serves up the images to the end user via 2 chained CDNs. We want to drop the middle CDN as it's not performing well and is a waste of money; in it's stead we plan to place a few reverse proxy web accelerators between the primary CDN and our image server.
You are aware then that a few reverse-proxy accelerators are in fact the
definition of a CDN? So you are building your own instead of paying for one.
Thank you for choosing Squid.
We currently recieve 152 hits/sec on average with about 550hps max to our secondary CDN from cache misses at the Primary.
I would like to serve a lot of this content straight from memory to get it out there as fast as possible.
I've read around that there are memory and processing limitations in Squid in the magnitude of 2-4GB RAM and 1 core/1 thread, respectively. So, my solution was to run multiple instances, as we don't have the rackspace to scale this out otherwise.
Memory limitations on large objects only exist in Squid-2. And 2-4GB RAM
issues reported recently are only due to 32-bit build + 32-bit hardware.
Your 8GB cache_mem settings below and stated object size show these are
not problems for your Squid.
152 req/sec is not enough to raise the CPU temperature with Squid, 550
might be noticeable but not a problem. 2700 req/sec has been measured in
accelerator Squid-2.6 on a 2.6GHz dual-core CPU and more performance
improvements have been added since then.
I've managed to build a working config of 1:1 squid:origin, but I am having trouble scaling this up and out.
Here is what I have attempted to do, maybe someone can point me in the right direction:
Current config:
User Browser -> Prim CDN -> Sec CDN -> Our Image server @ http port 80
New config idea:
User -> Prim CDN -> Squid0 @ http :80 -> round-robin to "parent" squid instances on same machine @ http :81, :82, etc -> Our Image server @ http :80
Squid0's (per diagram above) squid.conf:
acl Safe_ports port 80
acl PICS_DOM_COM dstdomain pics.domain.com
acl SQUID_PEERS src 127.0.0.1
http_access allow PICS_DOM_COM
icp_access allow SQUID_PEERS
miss_access allow SQUID_PEERS
http_port 80 accel defaultsite=pics.domain.com
cache_peer localhost parent 81 3130 name=imgCache1 round-robin proxy-only
cache_peer localhost parent 82 3130 name=imgCache2 round-robin proxy-only
cache_peer_access imgCache1 allow PICS_DOM_COM
cache_peer_access imgCache2 allow PICS_DOM_COM
cache_mem 8192 MB
maximum_object_size_in_memory 100 KB
cache_dir aufs /usr/local/squid0/cache 1024 16 256 -- This one isn't really relevant, as nothing is being cached on this instance (proxy-only)
icp_port 3130
visible_hostname pics.domain.com/0
Everything else is per the defaults in squid.conf.
"Parent" squids' (from above diagram) squid.conf:
acl Safe_ports port 81
acl PICS_DOM_COM dstdomain pics.domain.com
acl SQUID_PEERS src 127.0.0.1
http_access allow PICS_DOM_COM
icp_access allow SQUID_PEERS
miss_access allow SQUID_PEERS
http_port 81 accel defaultsite=pics.domain.com
cache_peer 192.168.0.223 parent 80 0 no-query originserver name=imgParent
cache_peer localhost sibling 82 3130 name=imgCache2 proxy-only
cache_peer_access imgParent allow PICS_DOM_COM
cache_peer_access imgCache2 allow PICS_DOM_COM
cache_mem 8192 MB
maximum_object_size_in_memory 100 KB
cache_dir aufs /usr/local/squid1/cache 10240 16 256
visible_hostname pics.domain.com/1
icp_port 3130
icp_hit_stale on
Everything else per defaults.
So, when I run this config and test I see the following happen in the logs:
From "Squid0" I see that it resolves to grab the image from one of it's parent caches. This is great! (some show as "Timeout_first_up_parent" and others as just "first_up_parent")
1237253713.769 62 127.0.0.1 TCP_MISS/200 2544 GET http://pics.domain.com:81/thumbnails/59/78/45673695.jpg - TIMEOUT_FIRST_UP_PARENT/imgParent image/jpeg
From the parent cache that it resolves to, I see that it grabs the image from IT'S parent, originserver (our image server). Subsequent requests are 'TCP_HIT' or mem hit. Great stuff!
1237253713.769 62 127.0.0.1 TCP_MISS/200 2694 GET http://pics.domain.com/thumbnails/59/78/45673695.jpg - FIRST_PARENT_MISS/imgCache1 image/jpeg
Problem is, it doesn't round-robin the requests to both of my "parent" squids and you end up with a very 1-sided cache. If I stop the "parent" instance that is resolving the items, the second "parent" doesn't take over either. If I then proceed to restart the "Squid0" instance, it will then direct the requests to the second "parent", but then the first wont recieve any requests. So I know both "parent" configs work, but I must be doing something wrong somewhere, or is this all just a silly idea...?
This is caused by Squid0 only sending ICP queries to a single peer
(itself?) on port 3130. Each squid needs a full set of its own unique
listening ports.
Can anyone comment on the best way to run a high-traffic set of accel cache instances similar to this, or how to fix what i've tried to do? Or another way to put a LOT of data into a squid instance's memory. (We have ~150Million x 2KB images that are randomly requested).
I'd like to see different content cached on each instance with little or no overlap with round-robin handling which squid gets to cache an item and icp handling which squid has that item.
I'm open to other ideas too..
Some things you have misunderstood:
cache_mem is the size of RAM-cache in used by each Squid
(same as a cache_dir but without disk parameters) so proxy-only
reduces its usefullness at the Squid0.
Your Squid0 should not need a cache_mem much larger than the max
hot-objects throughput for a short (minutes) window of data. I'd start
with around 500MB and test various changes.
Your Squid1/Squid2 should get as much as they can possibly hold.
Possibly with a heap replacement/garbage policy.
miss_access, is followed by implicit "deny all" but your stated topology
only the external CDN will be requesting access to Squid0. So this is
counter-productive. Leave it at the default 'allow all'.
On the hierarchy method itself I'd go with either a single Squid doing
everything (simpler and one should handle the peak load). Or if you must
expand to two layers use CARP to hash-balance the URL between the parent
peers.
round-robin will only balance the request count load. carp will balance
based on unique URI hashes so no object duplication occurs in the
second-layer Squid. It will also remove any need to do horizontal links
to the other sibling, since the carp hash guarantees that no other
2nd-layer squid has the object fresh.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE13
Current Beta Squid 3.1.0.6
