On 12.09.2012 10:54, Saurabh Sheth wrote:
Squid (versions: 3.1 and 2.6) has a object in its cache and responds to individual requests to this object just fine (TCP_HIT:NONE). From the access.log -> 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 41136 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 24752 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 28848 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 41136 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 24752 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 45232 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 28848 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 49328 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 49328 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 32944 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 37040 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:41:55 -0700] "GET http://originserver/data/object HTTP/1.1" 200 37040 TCP_HIT:NONE However, when I make a huge number of concurrent requests for the same object, squid fails to load the object from the disk fast enough and gives a TCP_SWAPFAIL_MISS -> 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 53424 TCP_HIT:NONE 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 37031 TCP_SWAPFAIL_MISS:DIRECT 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 28839 TCP_MISS:NONE 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 32935 TCP_MISS:NONE All subsequent requests hit the origin server directly causing huge load on the origin server (TCP_MISS:NONE) -> 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 32935 TCP_MISS:NONE 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 28839 TCP_MISS:NONE 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 37031 TCP_MISS:NONE 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 37031 TCP_MISS:NONE 10.192.x.x - - [11/Sep/2012:15:42:23 -0700] "GET http://originserver/data/object HTTP/1.1" 200 32935 TCP_MISS:NONE This is undesirable in the production setup, since such huge number of requests hitting the origin server directly have the result of a DOS attack on the origin server. This has brought down our origin server more than once now.
Well, when you think about it this is a DOS on Squid as well. The backend server is only facing the overflow which squid can't erase fast enough. So any attacker trying this has to pass *two* DOS thresholds, first the squid one then the backend on top. There is always another idiot infected PC, so DOS resolution is not about *solving* the traffic problem, but raising the bar and reducing the impact/damage when it happens.
I am looking for any help or pointers on how can I deal with such a huge number of concurrent requests to squid for the same object effectively, any help is highly appreciated. I am already considering the option of rate limiting using iptables, however if there is a effective way to deal with this in the squid configuration itself; I would love to understand.
You were a bit vague about which specific release versions of Squid you have. 2.6 should have had collapsed forwarding feature which acts as a great DOS barrier. It has not been ported to squid-3 yet, but efficiencies have been improved in the cache handling so you could try the latest 2.7 or 3.2 releases and see if this raises the bar high enough for you.
Amos
