Hello,
In case something hogs the workers, until you figure out what and solve it:
MaxConnectionsPerChild 0 # inifinite connections per child so apache
does not try to gracefully shut them down.
MaxSpareThreads = MaxRequestWorkers
Never do a graceful restart.
This is what I had to do in production with certain older version of
apache in which mod_weblogic would cause the same behaviour you
describe
Cheers
El lun, 28 nov 2022 a las 5:28, Mike Dewhirst
(<miked@xxxxxxxxxxxxxxx>) escribió:
>
> On 28/11/2022 12:09 pm, Mike Dewhirst wrote:
>
> On 26/11/2022 11:41 am, Mike Dewhirst wrote:
>
> On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
>
> Some script is probably hogging the workers and not letting them close so if you must, just restart, but judging by the amount of traffic you seem to have at the time I would truncate in logrotate and not restart or reload the server at all.
>
>
> I run a Python script on the production machine to delete the entire site (including directory structure) then redeploy everything from the repo whenever anything changes.
>
> On the basis that Apache (mod-wsgi) only reads the files once on reload, if they also had the dead finger of my script or perhaps the Python interpreter hanging on, that may well have "hogged" the workers.
>
> That script now restarts Apache instead of reload.
>
> I have now adjusted the logrotate script to avoid reload/restart logging status as follows:
>
> /var/log/apache2/*.log {
> daily
> missingok
> rotate 14
> compress
> delaycompress
> notifempty
> copytruncate
> sharedscripts
> prerotate
> if [ -d /etc/logrotate.d/httpd-prerotate ]; then
> run-parts /etc/logrotate.d/httpd-prerotate
> fi
> endscript
> postrotate
> if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
> invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
> fi
> endscript
> }
>
>
> ... and left the mpm-event settings as follows:
>
> <IfModule mpm_event_module>
> ServerLimit 32 # Thanks Eric Covener
> StartServers 16
> MinSpareThreads 400
> MaxSpareThreads 250
> ThreadLimit 64
> ThreadsPerChild 50
> AsyncRequestWorkerFactor 2
> MaxRequestWorkers 800
> MaxConnectionsPerChild 0
> </IfModule>
>
> I restarted the entire machine
>
> Next logrotate is 00:00 UTC -- I'll report tomorrow.
>
>
> Zero bytes in the error log since Saturday when I sent the above email.
>
> That covers the period with copytruncate in the Apache logrotate conf.
>
> I haven't updated the site so no restarts from me.
>
> I'll adjust the deployment script back to reload instead of restart before my next deployment and report back.
>
>
> Errors recommence as follows:
>
> ... not an error but the reload after a deployment script run...
>
> [Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... lengthy period ensues with no errors followed by another deployment reload ...
>
> [Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... another deployment script reload after a period with no errors ...
>
> [Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... but two seconds later ...
>
> [Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid 140638735415168] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> ... followed by more of the same at approximately 1 second intervals ...
>
> [Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid 140638735415168] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> About to revert my deployment script to restart Apache instead of reload ...
>
> Not sure what else I can do?
>
> Cheers
>
> Mike
>
>
>
> Cheers
>
> Mike
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|