Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit

[Mike Dewhirst <miked@xxxxxxxxxxxxxxx>]

On 28/11/2022 12:09 pm, Mike Dewhirst wrote:

On 26/11/2022 11:41 am, Mike Dewhirst wrote:

On 24/11/2022 6:48 pm, Daniel Ferradal wrote:

Some script is probably hogging the workers and not letting them close so if you must, just restart, but judging by the amount of traffic you seem to have at the time I would truncate in logrotate and not restart or reload the server at all.

I run a Python script on the production machine to delete the entire site (including directory structure) then redeploy everything from the repo whenever anything changes.

On the basis that Apache (mod-wsgi) only reads the files once on reload, if they also had the dead finger of my script or perhaps the Python interpreter hanging on, that may well have "hogged" the workers.

That script now restarts Apache instead of reload.

I have now adjusted the logrotate script to avoid reload/restart logging status as follows:

/var/log/apache2/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    copytruncate
    sharedscripts
    prerotate
    if [ -d /etc/logrotate.d/httpd-prerotate ]; then
        run-parts /etc/logrotate.d/httpd-prerotate
    fi
    endscript
    postrotate
    if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
        invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
    fi
    endscript
}


... and left the mpm-event settings as follows:

<IfModule mpm_event_module>
        ServerLimit                     32                 # Thanks Eric Covener
        StartServers                    16
        MinSpareThreads                 400
        MaxSpareThreads                 250
        ThreadLimit                     64
        ThreadsPerChild                 50
        AsyncRequestWorkerFactor        2
        MaxRequestWorkers               800
        MaxConnectionsPerChild          0
</IfModule>

I restarted the entire machine

Next logrotate is 00:00 UTC -- I'll report tomorrow.

Zero bytes in the error log since Saturday when I sent the above email.

That covers the period with copytruncate in the Apache logrotate conf.

I haven't updated the site so no restarts from me.

I'll adjust the deployment script back to reload instead of restart before my next deployment and report back.

    Errors recommence as follows:
    
    ... not an error but the reload after a deployment script run...
    
    [Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid
    140638735415168] AH00493: SIGUSR1 received.  Doing graceful restart
    [Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid
    140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
    mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
    [Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid
    140638735415168] AH00094: Command line: '/usr/sbin/apache2'
    
    ... lengthy period ensues with no errors followed by another
    deployment reload ...
    
    [Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid
    140638735415168] AH00493: SIGUSR1 received.  Doing graceful restart
    [Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid
    140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
    mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
    [Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid
    140638735415168] AH00094: Command line: '/usr/sbin/apache2'
    
    ... another deployment script reload after a period with no errors
    ...
    
    [Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid
    140638735415168] AH00493: SIGUSR1 received.  Doing graceful restart
    [Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid
    140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
    mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
    [Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid
    140638735415168] AH00094: Command line: '/usr/sbin/apache2'
    
    ... but two seconds later ...
    
    [Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid
    140638735415168] AH03490: scoreboard is full, not at
    MaxRequestWorkers.Increase ServerLimit.
    
    ... followed by more of the same at approximately 1 second intervals
    ...
    
    [Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid
    140638735415168] AH03490: scoreboard is full, not at
    MaxRequestWorkers.Increase ServerLimit.
    
    About to revert my deployment script to restart Apache instead of
    reload ...
    
    Not sure what else I can do?
    
    Cheers
    
    Mike
    
    
>  
> >         Cheers
> >         
> >         Mike
> >         
> > -- 
> > Signed email is an absolute defence against phishing. This email has
> > been signed with my private key. If you import my public key you can
> > automatically decrypt my signature and be sure it came from me. Just
> > ask and I'll send it to you. Your email software can handle signing.
> >       
>       
> -- 
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>     
    
-- 
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attachment: OpenPGP_signature
Description: OpenPGP digital signature