Re: Issue with Apache 2.4.51 hanging (Apache Users)

Hi Yann,

Just a quick follow up - we've tried removing mod_http2 but still managed to provoke a crash. See the error_log below when stopping/restarting after httpd becomes unresponsive. We need to be a bit more careful removing other modules to make sure they're not used, which is more time consuming - do you think this is still worth doing to address the issue?

If you have any other suggestions let me know.

Thanks.

Patrick

# cat /var/log/httpd/error_log
*** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x000055a67cc31e7f ***
*** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x000055a67cc31e7f ***
*** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x000055a67cc31e7f ***
*** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x000055a67cc31e7f ***
[Mon Oct 18 20:59:48.426225 2021] [core:notice] [pid 31207] AH00052: child pid 32036 exit signal Segmentation fault (11)
[Mon Oct 18 20:59:48.426389 2021] [core:notice] [pid 31207] AH00052: child pid 31246 exit signal Segmentation fault (11)
[Mon Oct 18 20:59:48.492282 2021] [core:notice] [pid 31207] AH00052: child pid 31253 exit signal Segmentation fault (11)
[Mon Oct 18 20:59:48.492312 2021] [core:notice] [pid 31207] AH00052: child pid 32289 exit signal Segmentation fault (11)
[Mon Oct 18 20:59:48.492455 2021] [mpm_prefork:notice] [pid 31207] AH00169: caught SIGTERM, shutting down
[Mon Oct 18 20:59:48.631928 2021] [suexec:notice] [pid 32620] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Oct 18 20:59:48.662384 2021] [lbmethod_heartbeat:notice] [pid 32626] AH02282: No slotmem from mod_heartmonitor
[Mon Oct 18 20:59:48.724408 2021] [mpm_prefork:notice] [pid 32626] AH00163: Apache/2.4.51 (Amazon) OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Mon Oct 18 20:59:48.724430 2021] [core:notice] [pid 32626] AH00094: Command line: '/usr/sbin/httpd'
[Mon Oct 18 20:59:49.724509 2021] [mpm_prefork:error] [pid 32626] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Patrick Verdon | Founder

Web: www.youreko.com

Mobile: +44 (0)7809 296438
Skype: patrick_verdon

This entire communication is sent on behalf of

Youreko Ltd and is strictly confidential to and

for the sole use of the intended addressee.

Registered in England - 7448349

On Mon, 18 Oct 2021 at 15:05, Patrick Verdon <patrick.verdon@xxxxxxxxxxx> wrote:

Hi Yann,

Many thanks for the super quick response. We'll try to remove mod_http2 and other modules as you suggest to see if that helps. I'll get back to you once we've had a chance to test it.

Thanks.

Patrick

--

Patrick Verdon | Founder
Web: www.youreko.com
Mobile: +44 (0)7809 296438
Skype: patrick_verdon

This entire communication is sent on behalf of
Youreko Ltd and is strictly confidential to and
for the sole use of the intended addressee.

Registered in England - 7448349

On Mon, 18 Oct 2021 at 12:57, Yann Ylavic <ylavic.dev@xxxxxxxxx> wrote:
Hi Patrick,

On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon
<patrick.verdon@xxxxxxxxxxx> wrote:
>
> # cat /var/log/httpd/error_log
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
[]
> *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x0000557f94567e4f ***
[]
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
> [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid 3166 exit signal Aborted (6)
[]
> [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: child pid 2741 exit signal Segmentation fault (11)
> *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x0000557f94567e4f ***
[]
> [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: child pid 2727 exit signal Aborted (6)
> [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] AH00169: caught SIGTERM, shutting down

The log seems to show a stop then start sequence (which is possibly
what "service httpd restart" does), anyway the stop crashes children
processes that at some point have reserved/handled mod_proxy
connections.

We will discuss whether/how to fix this on the dev@ mailing list, in
the meantime I'd suggest that:

> [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.

.. you do not "LoadModule http2_module mod_http2.so" in your MPM
prefork configuration, because due to its multithreaded nature (unlike
MPM prefork) mod_http2 implies that mod_proxy will have to
allocate/handle multiple simultaneous connection to the backend which
is what is causing the crash here.

> [Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] AH02282: No slotmem from mod_heartmonitor

Likewise you probably don't need lbmethod_heartbeat and several
modules in your list, so I'd suggest that you cleanup your LoadModules
a bit, ideally to the strict minimum needed.

Regards;
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx