Apache 2.4 timeouts/connection refused under high load. Works fine with version 2.2

[Fernando Marcelo <fernandomm.cpc@xxxxxxxxx>]

Hi,

I'm running Apache 2.4 (prefork) on a Centos 7 server. The server specs are pretty good: 2 x Xeon E5-2697 v3 @ 2.60GHz, 512GB of RAM and 8 SSDs in RAID 10.

This Apache server is used to run a PHP software and serve a few static files, although most of the static files are cached in our CDN service.

The normal load is between 8 and 15 but, due to the server specs, it's very responsive. Also, the number of requests/sec is usually below 300.

I noticed that sometimes Apache 2.4 will timeout or refuse connections even for local requests. This happens when the requests/sec is higher than 400 and/or the load is 20+.

Some notes:

1) The Apache error log doesn't show any errors at all. Increasing the verbosity of log also doesn't show anything related when the issue happens.

2) Even when this happens, the server is responsive. I can access it using SSH and other services works fine.

3) I have munin installed. When the issue happens at least 50% the RAM is free, CPU usage is ~70% and the I/O usage is very low.

I tried some tests changing lots of settings related to StartServers, MinSpareServers, Timeout etc. but nothing worked.

Then, I decided to downgrade to Apache 2.2 keeping the configuration as close as possible. I only changed what was really necessary to get 2.2 up. After 2 days the problem seems to be gone. I simply don't get more timeouts even with 800-900 requests/sec.

Any ideas about what could cause this issue?

Thanks!