Large DBM ssl_scache causing performance issues, possible DOS

["Mick Sheppard" <Mick.Sheppard@xxxxxxxxx>]

Hi,

 

I’ve been unable to find any information on the web concerning an issue we are currently having and want to check that we haven’t overlooked any possible options.

 

We run an instance of Apache 2.2 on Solaris 10 running as both a forward and reverse proxy for discrete SSL web services transactions. Our external clients authenticate themselves using SSL client certificates. We authenticate ourselves to external clients also using SSL client certificates. Our SSL session cache settings are as follows:

 

SSLSessionCache         dbm:/usr/local/apache2/logs/ssl_scache

SSLSessionCacheTimeout  300

 

We have seen increasing traffic over the past few weeks and observed a periodic slowdown in response time from the Apache server. A restart of the server has cleared the problem but it always came back after time. Using dtrace we were able to observe that a single httpd process was clocking up tens of thousands of reads in a short period of time, these were associated with the file /usr/local/apache2/logs/ssl_scache.pag. This file was quite large, over 1 GB, and the number of reads approximated to the file size/1024.

 

Looking at the debug output of Apache we could see that it was checking whether to expire entries from this file. Looking at the code each httpd process does at least one sequential read through the file every SSLSessionCacheTimeout. Whilst performing this read the SSL mutex is locked. Under load these scans appear to coincide for each child process with the result that the last client to gain the mutex will have been blocked for (time-to-scan-cache * nclients) seconds. This severely hampers the ability of our server to service requests.

 

As our web services traffic is in the form of discrete operations we don’t see that we have a need for the SSL session cache at all. Each request will be services by the initial client process so the other client processes have no need of the information. Consequently we are looking to change the SSLSessionCache setting to none to disable it.

 

We suspect that we are seeing this problem because of the use of client certificates and the fact that they seem to be stored in the cache. Based on our observations it would appear that if someone has a valid client cert then, with the ssl_scache configured in this manner, they can effectively perform a denial of service on the apache web server.

 

Our questions are whether anyone has come across this before and how they solved it, and also if there is will be any unforeseen result of us turning off the session cache?

 

 

Mick Sheppard

Enterprise Systems Support Analyst

CPP Group PLC

Holgate Park

York

YO26 4GA

Tel: 01904 544562

Fax:  01904 544683

Web: www.cpp.co.uk

Corporate: www.cppgroupplc.com

 

This is an email from the CPP Group Plc, Holgate Park, York, YO26 4GA; telephone +44 (0)1904 544500.
This message may contain information that is confidential. If you are not the intended recipient,
you may not peruse, use, disseminate, distribute or copy this message. If you have received this
message in error, please notify the sender immediately by email, facsimile or telephone and either
return or destroy the original message.

The CPP Group Plc accepts no responsibility for any changes made to this message after it has been
sent by the original author.  This email has been scanned for all viruses by the MessageLabs 
Email Security System.