I've been suffering from a significant memory leak on multiple servers
running Squid 3.5 for months, but was unable to reproduce it in a test
environment. I've now figured out how to reproduce it and have done
some investigation:
When using TPROXY, Squid generates fake "CONNECT 192.0.2.1:443"
requests, using the IP address that the client connected to. At
ssl_bump step 1, we peek and Squid generates another fake "CONNECT
example.com:443" request containing the SNI from the client's SSL handshake.
At ssl_bump step 2 we splice the connection and Squid does verification
to make sure that example.com does actually resolve to 192.0.2.1. If it
doesn't, Squid is supposed to reject the connection in
ClientRequestContext::hostHeaderVerifyFailed() to prevent clients from
manipulating the SNI to bypass ACLs.
Unfortunately, when verification fails, rather than actually dropping
the client's connection, Squid just leaves the client hanging.
Eventually the client (hopefully) times out and drops the connection
itself, but the associated ClientRequestContext is never destroyed.
This is testable by repeatedly executing:
openssl s_client -connect 17.252.76.30:443 -servername
courier.push.apple.com
That is a traffic pattern that we see in the real world and is now
clearly what is triggering the leak: Apple devices make connections to
addresses within the 17.0.0.0/8 network with an SNI of
"courier.push.apple.com". courier.push.apple.com resolves to a CNAME
pointing to courier-push-apple.com.akadns.net, but
courier-push-apple.com.akadns.net doesn't exist. Since Squid can't
verify the connection, it won't allow it and after 30 seconds the client
times out. Each Apple device keeps retrying the connection, leaking a
ClientRequestContext each time, and before long we've leaked several
gigabytes of memory (on some networks I'm seeing 16GB or more of leaked
RAM over 24 hours!).
Unfortunately I'm a bit lost in the Squid code and can't quite figure
out how to gracefully terminate the connection and destroy the context.
--
- Steve Hill
Technical Director
Opendium Limited http://www.opendium.com
Sales / enquiries:
Email: sales@xxxxxxxxxxxx
Phone: +44-1792-824568 / sip:sales@xxxxxxxxxxxx
Support:
Email: support@xxxxxxxxxxxx
Phone: +44-1792-825748 / sip:support@xxxxxxxxxxxx
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
