Hi there,
I'm running some PHP-based sites on CentOS 8, which uses version 2.4.37 of Apache. Apache is set to use MPM Event mode, which is also the default.
Once per day during a quiet period, logrotate rotates the Apache logs by doing a graceful restart (reload) as part of a postrotate command:
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
This causes the following line to appear in the Apache error log, which I believe to be normal and expected:
[mpm_event:notice] [pid 1413:tid 140292271810880] AH00493: SIGUSR1 received. Doing graceful restart
The log is then rotated, and in the newly created error log, I see the following:
[lbmethod_heartbeat:notice] [pid 1413:tid 140292271810880] AH02282: No slotmem from mod_heartmonitor
[mpm_event:notice] [pid 1413:tid 140292271810880] AH00489: Apache/2.4.37 (centos) OpenSSL/1.1.1g configured -- resuming normal operations
[core:notice] [pid 1413:tid 140292271810880] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
Then, immediately after, I see this error repeated once per second:
[mpm_event:error] [pid 1413:tid 140292271810880] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
The server then becomes unresponsive and will not answer HTTPS requests. It continues logging the above error and will not begin working again until I manually restart the Apache process.
All configs on this server are essentially the default for CentOS 8. I have not tried making any adjustments to the MPM Event config settings -- mostly because I'm not sure what setting for ServerLimit would be "correct," and also because this problem happens *only* when logrotate does its graceful restart, and not during other periods of high traffic.
My research on this issue brings up a lot of results about an old Apache bug related to MPM Event mode, but my version of Apache shouldn't be affected by this particular bug. I have also seen suggestions that some long-running processes may hang and eat up all available worker processes, but I don't understand why this would interrupt the restart. And I don't have any particularly long-running processes served by Apache -- just quickly executing PHP scripts.
Can anyone suggest what might fix this issue, given that it seems to happen only during a graceful reload? Should I try changing the method that logrotate uses to restart Apache?
Is there some other way I can avoid the server hanging, or is it really necessary to increase the ServerLimit as the error suggests?
Thank you very much for any insight or help.
