We've got squid accelerator setups, and trying to get away from using layer7 balancers for URL hashing, so we thought we'd use CARP.
The basics are: a pool of 18 servers, balanced round-robin. We have CARP squid instances on port 80, a caching squid on 81.
Our carp cache_peer lines look like:
cache_peer 69.147.123.121 parent 81 7 carp no-query name=photocache201 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.122 parent 81 7 carp no-query name=photocache202 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.123 parent 81 7 carp no-query name=photocache203 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.124 parent 81 7 carp no-query name=photocache204 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.125 parent 81 7 carp no-query name=photocache205 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.126 parent 81 7 carp no-query name=photocache206 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.32 parent 81 7 carp no-query name=photocache207 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.33 parent 81 7 carp no-query name=photocache208 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.34 parent 81 7 carp no-query name=photocache209 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.35 parent 81 7 carp no-query name=photocache210 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.36 parent 81 7 carp no-query name=photocache211 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.37 parent 81 7 carp no-query name=photocache212 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.38 parent 81 7 carp no-query name=photocache213 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.39 parent 81 7 carp no-query name=photocache214 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.40 parent 81 7 carp no-query name=photocache215 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.41 parent 81 7 carp no-query name=photocache216 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.42 parent 81 7 carp no-query name=photocache217 monitorurl=/index.htm monitorinterval=60
cache_peer 69.147.123.43 parent 81 7 carp no-query name=photocache218 monitorurl=/index.htm monitorinterval=60
The carp instances get an equal balance from the load balancer, but we don't see an equal balance across the caching
squid instances. Instead, we see a distribution that looks exactly like the sequence of hosts listed as cache_peers. For
example, photocache201 gets the most requests, and it decreases down the line, and photocache218 gets the *least* requests.
Is this expected? How can we get a real balance?
The cache manager confirms what we're seeing:
$squidclient -p 80 cache_object://127.0.0.1/carp
HTTP/1.0 200 OK
Server: squid/2.7.STABLE5
Date: Fri, 19 Dec 2008 18:52:21 GMT
Content-Type: text/plain
Expires: Fri, 19 Dec 2008 18:52:21 GMT
X-Cache: MISS from photocache201.flickr
X-Cache-Lookup: MISS from photocache201.flickr
Via: 1.0 photocache201.flickr (squid/2.7.STABLE5)
Connection: close
Hostname Hash Multiplier Factor Actual
apache_peer 0 0.000000 0.000000 0.007863
photocache201 b7d71c0d 1.000000 0.055556 0.162550
photocache202 e4836670 1.000000 0.055556 0.133271
photocache203 114fb0d4 1.000000 0.055556 0.072396
photocache204 3e1bfb37 1.000000 0.055556 0.076387
photocache205 6ac8459a 1.000000 0.055556 0.064102
photocache206 97948ffd 1.000000 0.055556 0.045340
photocache207 c440da60 1.000000 0.055556 0.058199
photocache208 f10d24c3 1.000000 0.055556 0.039018
photocache209 1dd96f27 1.000000 0.055556 0.050036
photocache210 b7d0820d 1.000000 0.055556 0.043762
photocache211 e49ccc70 1.000000 0.055556 0.038504
photocache212 114916d4 1.000000 0.055556 0.032422
photocache213 3e156137 1.000000 0.055556 0.031728
photocache214 6ac1ab9a 1.000000 0.055556 0.026673
photocache215 978df5fd 1.000000 0.055556 0.028562
photocache216 c45a4060 1.000000 0.055556 0.030378
photocache217 f1068ac3 1.000000 0.055556 0.024259
photocache218 1dd2d527 1.000000 0.055556 0.034549
(the apache_peer is a local peer on each box for origin server healthchecks, it can be ignored)
To confirm this, we even reversed the cache_peer lines; it results in photocache218 getting the most, photocache201 getting
the least. :)
What gives?
--john
