Search squid archive

Large memory leak with ssl_peek (now partly understood)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




I've been suffering from a significant memory leak on multiple servers running Squid 3.5 for months, but was unable to reproduce it in a test environment. I've now figured out how to reproduce it and have done some investigation:

When using TPROXY, Squid generates fake "CONNECT 192.0.2.1:443" requests, using the IP address that the client connected to. At ssl_bump step 1, we peek and Squid generates another fake "CONNECT example.com:443" request containing the SNI from the client's SSL handshake.

At ssl_bump step 2 we splice the connection and Squid does verification to make sure that example.com does actually resolve to 192.0.2.1. If it doesn't, Squid is supposed to reject the connection in ClientRequestContext::hostHeaderVerifyFailed() to prevent clients from manipulating the SNI to bypass ACLs.

Unfortunately, when verification fails, rather than actually dropping the client's connection, Squid just leaves the client hanging. Eventually the client (hopefully) times out and drops the connection itself, but the associated ClientRequestContext is never destroyed.

This is testable by repeatedly executing:
openssl s_client -connect 17.252.76.30:443 -servername courier.push.apple.com

That is a traffic pattern that we see in the real world and is now clearly what is triggering the leak: Apple devices make connections to addresses within the 17.0.0.0/8 network with an SNI of "courier.push.apple.com". courier.push.apple.com resolves to a CNAME pointing to courier-push-apple.com.akadns.net, but courier-push-apple.com.akadns.net doesn't exist. Since Squid can't verify the connection, it won't allow it and after 30 seconds the client times out. Each Apple device keeps retrying the connection, leaking a ClientRequestContext each time, and before long we've leaked several gigabytes of memory (on some networks I'm seeing 16GB or more of leaked RAM over 24 hours!).

Unfortunately I'm a bit lost in the Squid code and can't quite figure out how to gracefully terminate the connection and destroy the context.

--
 - Steve Hill
   Technical Director
   Opendium Limited     http://www.opendium.com

Sales / enquiries:
   Email:            sales@xxxxxxxxxxxx
   Phone:            +44-1792-824568 / sip:sales@xxxxxxxxxxxx

Support:
   Email:            support@xxxxxxxxxxxx
   Phone:            +44-1792-825748 / sip:support@xxxxxxxxxxxx
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users




[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux