On 10/03/11 00:04, Paul wrote:
In the last 24 hours I've started seeing thousands of requests to my LDAP server being sent by the squid_ldap_auth helper. In my cache.log I'm seeing hundreds of "squid_ldap_auth: WARNING, LDAP search error 'Can't contact LDAP server'" entries, interspersed with "2011/03/09 10:49:29| commBind: Cannot bind socket FD 76 to *:0: (98) Address already in use". The CPU usage on my LDAP sever is extremely high and this is obviously causing problem for my users
The "squid_ldap_auth:" lines are coming from the helper. The problems is exactly as stated, the LDAP server is not answering connection requests.
The "commBind:" lines are from squid itself. Squid-2 always uses bind(), even if there is no address being bound. That message indicates there is no socket available to be dedicated on the link or the stack is getting confused.
It seems like your kernel or networking is not able to cope with the number of TCP sockets those thousands of requests are needing to use.
tcpdump shows the requests going to the LDAP server have no "user" information i.e cn..none.*..groupMembership..cn=InternetAccess,o=org and that for each request to LDAP there is NO corresponding request to Squid. It's as if a process on one of my internal machines is sending a request in such a way that the squid_ldap_auth helper is getting stuck yet I can't see this in the tcpdump trace either.
Check some of the HTTP headers arriving into Squid. Base-64 decoding the "random" letter string on the Proxy-Authorization: should come on up with "username:password". If the username is actually missing it is probably malicious.
Reloading or restarting Squid relieves the problem for a short while but it soon reoccurs I'm using Squid 2.7Stable6-6.1 on openSuSE_11.3 64 bit with all modules up to date from the official SuSE repos. Squid is a forward proxy only and there is nothing suspicious coming from the Internet at large
For these auth symptoms on a forward proxy it would be suspicious stuff coming out of the LAN to look for. Infected clients, broken software becoming popular, etc.
Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
