Re: Issue with Apache 2.4.51 hanging (Apache Users)

Hi Daniel,

Thanks for the feedback.

Unfortunately that's not really an option for us, as it's a major architecture change, and is only something we would consider as a last resort.

Using mod_php works well for us and our configuration/application has not changed in years - the crashes started very specifically when we upgraded to Apache 2.4.48 from 2.4.46.

From what I can see, mod_php is still used widely and I assume it is therefore still a supported module?

I'd be grateful for any other suggestions on how to get to the bottom of the issue.

Thanks.

Patrick

Patrick Verdon | Founder

Web: www.youreko.com

Mobile: +44 (0)7809 296438
Skype: patrick_verdon

This entire communication is sent on behalf of

Youreko Ltd and is strictly confidential to and

for the sole use of the intended addressee.

Registered in England - 7448349

On Tue, 26 Oct 2021 at 09:56, Daniel Ferradal <dferradal@xxxxxxxxxx> wrote:

Hello,

Seems it is related to third party php module being loaded in Apache HTTPD, no?

It would be awesome if you can move your php files to be dealt with php-fpm instead and from apache just proxy to php-fpm.. and see if your Apache server would ever hang again (if you do it you could even use mpm_event instead of prefork to have http/2 working).

Cheers

El vie, 22 oct 2021 a las 18:16, Patrick Verdon (<patrick.verdon@xxxxxxxxxxx>) escribió:
Hi Yann,

I finally managed to provoke the crash - there were 6 core dumps, I followed your instructions and have pasted the gdb output below for each one.

Let me know if this sheds any light on the problem.

Thanks.

Patrick

--

# gdb /usr/sbin/httpd /tmp/core.1221
Thread 2 (Thread 0x7f5876bff840 (LWP 1221)):
#0 0x00007f587518e687 in kill () from /lib64/libc.so.6
#1 <signal handler called>
#2 0x00007f5861def0d8 in ?? () from /lib64/libgcc_s.so.1
#3 0x00007f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#4 0x00007f587526cbf6 in backtrace () from /lib64/libc.so.6
#5 0x00007f58751d0fb4 in __libc_message () from /lib64/libc.so.6
#6 0x00007f58751d8854 in malloc_consolidate () from /lib64/libc.so.6
#7 0x00007f58751d923e in _int_free () from /lib64/libc.so.6
#8 0x00007f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#9 0x00007f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#10 0x00007f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#11 <signal handler called>
#12 0x00007f587524bcfd in poll () from /lib64/libc.so.6
#13 0x00007f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so
#14 0x00007f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so
#15 0x00007f586807ab3e in _php_stream_fill_read_buffer () from /etc/httpd/modules/libphp-7.0.so
#16 0x00007f586807afbb in _php_stream_get_line () from /etc/httpd/modules/libphp-7.0.so
#17 0x00007f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so
#18 0x00007f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so
#19 0x00007f586807d079 in _php_stream_open_wrapper_ex () from /etc/httpd/modules/libphp-7.0.so
#20 0x00007f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so
#21 0x00007f585e957b3c in ?? ()
#22 0x00007f5866fcde00 in ?? ()
#23 0x0000000000001c07 in ?? ()
#24 0x00007f585605b380 in ?? ()
#25 0x00007f585605b3a0 in ?? ()
#26 0x00007f5854859bd8 in ?? ()
#27 0x00007f5866fefe00 in ?? ()
#28 0x00007f5853291dc0 in ?? ()
#29 0x00007f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so
#30 0x00007f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so
#31 0x00007f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so
#32 0x00007f58680fe9eb in execute_ex () from /etc/httpd/modules/libphp-7.0.so
#33 0x00007f586814849f in zend_execute () from /etc/httpd/modules/libphp-7.0.so
#34 0x00007f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/libphp-7.0.so
#35 0x00007f5868064808 in php_execute_script () from /etc/httpd/modules/libphp-7.0.so
#36 0x00007f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so
#37 0x000055e5af703bc0 in ap_run_handler ()
#38 0x000055e5af704109 in ap_invoke_handler ()
#39 0x000055e5af71a3ea in ap_process_async_request ()
#40 0x000055e5af71a6be in ap_process_request ()
#41 0x000055e5af716764 in ?? ()
#42 0x000055e5af70d810 in ap_run_process_connection ()
#43 0x00007f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#44 0x00007f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#45 0x00007f586c277bc6 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#46 0x000055e5af6e3e8e in ap_run_mpm ()
#47 0x000055e5af6dc0b9 in main ()

Thread 1 (Thread 0x7f5853d61700 (LWP 1222)):
#0 0x00007f587524da63 in select () from /lib64/libc.so.6
#1 0x00007f587596ff65 in apr_sleep () from /usr/lib64/libapr-1.so.0
#2 0x00007f586d1f8e62 in ?? () from /etc/httpd/modules/mod_watchdog.so
#3 0x00007f5875731e75 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f5875256a2d in clone () from /lib64/libc.so.6

# gdb /usr/sbin/httpd /tmp/core.1957
Thread 1 (Thread 0x7f5876bff840 (LWP 1957)):
#0 0x00007f5861def0d8 in ?? () from /lib64/libgcc_s.so.1
#1 0x00007f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2 0x00007f587526cbf6 in backtrace () from /lib64/libc.so.6
#3 0x00007f58751d0fb4 in __libc_message () from /lib64/libc.so.6
#4 0x00007f58751d8854 in malloc_consolidate () from /lib64/libc.so.6
#5 0x00007f58751d923e in _int_free () from /lib64/libc.so.6
#6 0x00007f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#7 0x00007f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#8 0x00007f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#9 <signal handler called>
#10 0x00007f587524bcfd in poll () from /lib64/libc.so.6
#11 0x00007f5865f45ea9 in ?? ()
#12 0x000055e500d80001 in ?? ()
#13 0x000000000006adf5 in ?? ()
#14 0x00000000000bf117 in ?? ()
#15 0x000000012ea5bfb4 in ?? ()
#16 0x0000000000000000 in ?? ()

# gdb /usr/sbin/httpd /tmp/core.29826
Thread 1 (Thread 0x7f5876bff840 (LWP 29826)):
#0 0x00007f5861def0d8 in ?? () from /lib64/libgcc_s.so.1
#1 0x00007f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2 0x00007f587526cbf6 in backtrace () from /lib64/libc.so.6
#3 0x00007f58751d0fb4 in __libc_message () from /lib64/libc.so.6
#4 0x00007f58751d8854 in malloc_consolidate () from /lib64/libc.so.6
#5 0x00007f58751d923e in _int_free () from /lib64/libc.so.6
#6 0x00007f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#7 0x00007f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#8 0x00007f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#9 <signal handler called>
#10 0x00007f587524bcfd in poll () from /lib64/libc.so.6
#11 0x00007f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so
#12 0x00007f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so
#13 0x00007f586807ab3e in _php_stream_fill_read_buffer () from /etc/httpd/modules/libphp-7.0.so
#14 0x00007f586807afbb in _php_stream_get_line () from /etc/httpd/modules/libphp-7.0.so
#15 0x00007f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so
#16 0x00007f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so
#17 0x00007f586807d079 in _php_stream_open_wrapper_ex () from /etc/httpd/modules/libphp-7.0.so
#18 0x00007f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so
#19 0x00007f585e957b3c in ?? ()
#20 0x00007f5850fea2d8 in ?? ()
#21 0x0000000000001c07 in ?? ()
#22 0x00007f585605b380 in ?? ()
#23 0x00007f585605b3a0 in ?? ()
#24 0x00007f5854859bd8 in ?? ()
#25 0x00007f5850fe9760 in ?? ()
#26 0x00007f5850fdcb80 in ?? ()
#27 0x00007f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so
#28 0x00007f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so
#29 0x00007f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so
#30 0x00007f58680fe9eb in execute_ex () from /etc/httpd/modules/libphp-7.0.so
#31 0x00007f586814849f in zend_execute () from /etc/httpd/modules/libphp-7.0.so
#32 0x00007f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/libphp-7.0.so
#33 0x00007f5868064808 in php_execute_script () from /etc/httpd/modules/libphp-7.0.so
#34 0x00007f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so
#35 0x000055e5af703bc0 in ap_run_handler ()
#36 0x000055e5af704109 in ap_invoke_handler ()
#37 0x000055e5af71a3ea in ap_process_async_request ()
#38 0x000055e5af71a6be in ap_process_request ()
#39 0x000055e5af716764 in ?? ()
#40 0x000055e5af70d810 in ap_run_process_connection ()
#41 0x00007f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#42 0x00007f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#43 0x00007f586c277bc6 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#44 0x000055e5af6e3e8e in ap_run_mpm ()
#45 0x000055e5af6dc0b9 in main ()

# gdb /usr/sbin/httpd /tmp/core.29938
Thread 1 (Thread 0x7f5876bff840 (LWP 29938)):
#0 0x00007f5861def0d8 in ?? () from /lib64/libgcc_s.so.1
#1 0x00007f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2 0x00007f587526cbf6 in backtrace () from /lib64/libc.so.6
#3 0x00007f58751d0fb4 in __libc_message () from /lib64/libc.so.6
#4 0x00007f58751d8854 in malloc_consolidate () from /lib64/libc.so.6
#5 0x00007f58751d923e in _int_free () from /lib64/libc.so.6
#6 0x00007f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#7 0x00007f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#8 0x00007f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#9 <signal handler called>
#10 0x00007f587524da63 in select () from /lib64/libc.so.6
#11 0x00007f58661a0bf9 in ?? ()
#12 0x00007ffeba76e7d8 in ?? ()
#13 0x0000004b00000000 in ?? ()
#14 0x00007f5866e17c40 in ?? ()
#15 0x4024000000000000 in ?? ()
#16 0x0000000000000009 in ?? ()
#17 0x00000000000e8e93 in ?? ()
#18 0xfffe000000000000 in ?? ()
#19 0x0000000000000fff in ?? ()
#20 0x0000000000000000 in ?? ()

# gdb /usr/sbin/httpd /tmp/core.30105
Thread 1 (Thread 0x7f5876bff840 (LWP 30105)):
#0 0x00007f5861def0d8 in ?? () from /lib64/libgcc_s.so.1
#1 0x00007f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2 0x00007f587526cbf6 in backtrace () from /lib64/libc.so.6
#3 0x00007f58751d0fb4 in __libc_message () from /lib64/libc.so.6
#4 0x00007f58751d8854 in malloc_consolidate () from /lib64/libc.so.6
#5 0x00007f58751d923e in _int_free () from /lib64/libc.so.6
#6 0x00007f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#7 0x00007f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#8 0x00007f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#9 <signal handler called>
#10 0x00007f587524da63 in select () from /lib64/libc.so.6
#11 0x00007f58661a0bf9 in ?? ()
#12 0x00007ffeba76e7d8 in ?? ()
#13 0x0000002400000000 in ?? ()
#14 0x00007f5866e17c40 in ?? ()
#15 0x4024000000000000 in ?? ()
#16 0x0000000000000009 in ?? ()
#17 0x00000000000e7588 in ?? ()
#18 0x0000001fe0000000 in ?? ()
#19 0x0000000000000000 in ?? ()

# gdb /usr/sbin/httpd /tmp/core.31389
Thread 1 (Thread 0x7f5876bff840 (LWP 31389)):
#0 0x00007f5861def0d8 in ?? () from /lib64/libgcc_s.so.1
#1 0x00007f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2 0x00007f587526cbf6 in backtrace () from /lib64/libc.so.6
#3 0x00007f58751d0fb4 in __libc_message () from /lib64/libc.so.6
#4 0x00007f58751d8854 in malloc_consolidate () from /lib64/libc.so.6
#5 0x00007f58751d923e in _int_free () from /lib64/libc.so.6
#6 0x00007f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0
#7 0x00007f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#8 0x00007f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#9 <signal handler called>
#10 0x00007f587524bcfd in poll () from /lib64/libc.so.6
#11 0x00007f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so
#12 0x00007f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so
#13 0x00007f586807ab3e in _php_stream_fill_read_buffer () from /etc/httpd/modules/libphp-7.0.so
#14 0x00007f586807afbb in _php_stream_get_line () from /etc/httpd/modules/libphp-7.0.so
#15 0x00007f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so
#16 0x00007f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so
#17 0x00007f586807d079 in _php_stream_open_wrapper_ex () from /etc/httpd/modules/libphp-7.0.so
#18 0x00007f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so
#19 0x00007f585e957b3c in ?? ()
#20 0x00007f5852ea4ab8 in ?? ()
#21 0x0000000000001c07 in ?? ()
#22 0x00007f585605b380 in ?? ()
#23 0x00007f585605b3a0 in ?? ()
#24 0x00007f5854859bd8 in ?? ()
#25 0x00007f5852e0d2a0 in ?? ()
#26 0x00007f5852e99740 in ?? ()
#27 0x00007f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so
#28 0x00007f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so
#29 0x00007f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so
#30 0x00007f58680fe9eb in execute_ex () from /etc/httpd/modules/libphp-7.0.so
#31 0x00007f586814849f in zend_execute () from /etc/httpd/modules/libphp-7.0.so
#32 0x00007f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/libphp-7.0.so
#33 0x00007f5868064808 in php_execute_script () from /etc/httpd/modules/libphp-7.0.so
#34 0x00007f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so
#35 0x000055e5af703bc0 in ap_run_handler ()
#36 0x000055e5af704109 in ap_invoke_handler ()
#37 0x000055e5af71a3ea in ap_process_async_request ()
#38 0x000055e5af71a6be in ap_process_request ()
#39 0x000055e5af716764 in ?? ()
#40 0x000055e5af70d810 in ap_run_process_connection ()
#41 0x00007f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#42 0x00007f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#43 0x00007f586c277bc6 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#44 0x000055e5af6e3e8e in ap_run_mpm ()
#45 0x000055e5af6dc0b9 in main ()

--

Patrick Verdon | Founder
Web: www.youreko.com
Mobile: +44 (0)7809 296438
Skype: patrick_verdon

This entire communication is sent on behalf of
Youreko Ltd and is strictly confidential to and
for the sole use of the intended addressee.

Registered in England - 7448349

On Fri, 22 Oct 2021 at 10:53, Deepak Goel <deicool@xxxxxxxxx> wrote:
We will then have to look into what is happening in the step (probably add debugging code):

Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child process at the end to release the memory, using posix_kill(PID, 28).

Deepak
"The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi"

+91 73500 12833
deicool@xxxxxxxxx

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home

On Fri, Oct 22, 2021 at 3:07 PM Patrick Verdon <patrick.verdon@xxxxxxxxxxx> wrote:
Correct.

On Fri, 22 Oct 2021 at 10:35, Deepak Goel <deicool@xxxxxxxxx> wrote:
I guess what you are saying is that the following error happens during startup and not during normal operation

( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting)

Deepak
"The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi"

+91 73500 12833
deicool@xxxxxxxxx

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home

On Fri, Oct 22, 2021 at 2:23 PM Patrick Verdon <patrick.verdon@xxxxxxxxxxx> wrote:
Hi Yann,

Quick update - we've enabled the core dumps but haven't been able to reproduce the issue. After removing mod_http2 the first time we were able to trigger the crash after 14 attempts but we've since tried over 100 times with no luck. We'll keep trying as there's nothing worse than knowing there's a bug lurking that can cause a crash.

@Deepak - thanks for your suggestion but hitting MaxRequestWorkers is a quirk of our installation, we load the max workers on startup so that the PHP application is primed and ready, rather than have Apache spawn lots of heavy processes. This is the same configuration we've had for years without ever experiencing Apache hanging until the upgrade to 2.4.48.

Thanks.

Patrick

--

Patrick Verdon | Founder
Web: www.youreko.com
Mobile: +44 (0)7809 296438
Skype: patrick_verdon

This entire communication is sent on behalf of
Youreko Ltd and is strictly confidential to and
for the sole use of the intended addressee.

Registered in England - 7448349

On Tue, 19 Oct 2021 at 11:00, Deepak Goel <deicool@xxxxxxxxx> wrote:
Hi

Looks like the step 2 in your process is not working in the upgraded version of apache.

Therefore it is vomiting out the following:
server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Deepak
"The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi"

+91 73500 12833
deicool@xxxxxxxxx

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home

On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon <patrick.verdon@xxxxxxxxxxx> wrote:
Hi All,

I'd appreciate some feedback on an issue I'm experiencing. I've spent quite some time researching the problem as it causes a serious outage in our application. I've searched the Web, Stack Overflow, this list's mail archives, the latest Apache bugs, and more, but have not been able to find any reports of a similar issue.

Background. I'm running the latest Apache 2.4.51 on Amazon Linux with mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our application take about 45 seconds to complete so there is a warm-up cache procedure at regular intervals during the day which primes the varnish cache. The following steps reliably cause Apache to hang, requiring a manual restart:
Varnish cache is cleared, causing spike in load on httpd
Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child process at the end to release the memory, using posix_kill(PID, 28).
Apache hangs and does not recover. Varnish serves 503s.
Manual restart required: service httpd restart
Errors in the log show that 2 children had segmentation faults, presumably the 2 with long running processes.

Albeit ugly, this process has been running for a year and a half without any issues. We traced the date that crashes started to the date Apache was upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an issue in 2.4.51.

See the error_log below and details about the installation.

Any feedback on where to report this issue would be much appreciated.

Thanks.

Patrick

--

# cat /var/log/httpd/error_log
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
*** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x0000557f94567e4f ***
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed.
[Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid 3166 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: child pid 3483 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: child pid 2657 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: child pid 2660 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052: child pid 2661 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990579 2021] [core:notice] [pid 2620] AH00052: child pid 3172 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990592 2021] [core:notice] [pid 2620] AH00052: child pid 2681 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990603 2021] [core:notice] [pid 2620] AH00052: child pid 3254 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990615 2021] [core:notice] [pid 2620] AH00052: child pid 2685 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990627 2021] [core:notice] [pid 2620] AH00052: child pid 2688 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990639 2021] [core:notice] [pid 2620] AH00052: child pid 3015 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990652 2021] [core:notice] [pid 2620] AH00052: child pid 2696 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990664 2021] [core:notice] [pid 2620] AH00052: child pid 2699 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990680 2021] [core:notice] [pid 2620] AH00052: child pid 2710 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990692 2021] [core:notice] [pid 2620] AH00052: child pid 2713 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990703 2021] [core:notice] [pid 2620] AH00052: child pid 3250 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990716 2021] [core:notice] [pid 2620] AH00052: child pid 2721 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990726 2021] [core:notice] [pid 2620] AH00052: child pid 2724 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990739 2021] [core:notice] [pid 2620] AH00052: child pid 2734 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990750 2021] [core:notice] [pid 2620] AH00052: child pid 3471 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990769 2021] [core:notice] [pid 2620] AH00052: child pid 3109 exit signal Aborted (6)
[Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: child pid 2741 exit signal Segmentation fault (11)
*** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x0000557f94567e4f ***
[Sun Oct 17 15:53:48.056539 2021] [core:notice] [pid 2620] AH00052: child pid 3019 exit signal Aborted (6)
[Sun Oct 17 15:53:48.056584 2021] [core:notice] [pid 2620] AH00052: child pid 2707 exit signal Segmentation fault (11)
[Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: child pid 2727 exit signal Aborted (6)
[Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] AH00169: caught SIGTERM, shutting down
[Sun Oct 17 15:53:48.151770 2021] [suexec:notice] [pid 3575] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] AH02282: No slotmem from mod_heartmonitor
[Sun Oct 17 15:53:48.243891 2021] [mpm_prefork:notice] [pid 3581] AH00163: Apache/2.4.51 (Amazon) OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Sun Oct 17 15:53:48.243923 2021] [core:notice] [pid 3581] AH00094: Command line: '/usr/sbin/httpd'
[Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

# httpd -V
Server version: Apache/2.4.51 (Amazon)
Server built: Oct 8 2021 19:30:47
Server's Module Magic Number: 20120211:118
Server loaded: APR 1.6.3, APR-UTIL 1.5.4
Compiled using: APR 1.6.3, APR-UTIL 1.5.4
Architecture: 64-bit
Server MPM: prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

# httpd -M
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_socache_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cache_disk_module (shared)
cache_socache_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
echo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
headers_module (shared)
http2_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
logio_module (shared)
macro_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
request_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
socache_dbm_module (shared)
socache_memcache_module (shared)
socache_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
suexec_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
watchdog_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_prefork_module (shared)
proxy_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
lbmethod_heartbeat_module (shared)
proxy_ajp_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_hcheck_module (shared)
proxy_scgi_module (shared)
proxy_uwsgi_module (shared)
proxy_wstunnel_module (shared)
ssl_module (shared)
cgi_module (shared)
php7_module (shared)

# yum list | grep mod_
lighttpd-mod_authn_gssapi.x86_64 1.4.53-1.36.amzn1 amzn-updates
lighttpd-mod_authn_mysql.x86_64 1.4.53-1.36.amzn1 amzn-updates
lighttpd-mod_authn_pam.x86_64 1.4.53-1.36.amzn1 amzn-updates
lighttpd-mod_geoip.x86_64 1.4.53-1.36.amzn1 amzn-updates
lighttpd-mod_mysql_vhost.x86_64 1.4.53-1.36.amzn1 amzn-updates
mod_auth_kerb.x86_64 5.4-10.9.amzn1 amzn-main
mod_auth_mellon.x86_64 0.13.1-1.6.amzn1 amzn-updates
mod_auth_mysql.x86_64 1:3.0.0-18.10.amzn1 amzn-main
mod_auth_pgsql.x86_64 2.0.3-10.1.5.amzn1 amzn-main
mod_authz_ldap.x86_64 0.26-16.8.amzn1 amzn-main
mod_dav_svn.x86_64 1.9.7-1.54.amzn1 amzn-main
mod_fcgid.x86_64 2.3.9-1.6.amzn1 amzn-main
mod_geoip.x86_64 1.2.7-1.2.amzn1 amzn-main
mod_nss.x86_64 1.0.10-1.13.amzn1 amzn-main
mod_perl.x86_64 2.0.7-7.28.amzn1 amzn-updates
mod_perl-devel.x86_64 2.0.7-7.28.amzn1 amzn-updates
mod_proxy_html.x86_64 3.1.2-7.3.amzn1 amzn-main
mod_python26.x86_64 3.3.1-17.20.amzn1 amzn-main
mod_python27.x86_64 3.3.1-17.20.amzn1 amzn-main
mod_security.x86_64 2.8.0-5.27.amzn1 amzn-main
mod_security_crs.noarch 2.2.8-2.5.amzn1 amzn-main
mod_security_crs-extras.noarch 2.2.8-2.5.amzn1 amzn-main
mod_ssl.x86_64 1:2.2.34-1.16.amzn1 amzn-main
mod_wsgi-python26.x86_64 3.2-6.12.amzn1 amzn-updates
mod_wsgi-python27.x86_64 3.2-6.12.amzn1 amzn-updates

--

Patrick Verdon | Founder
Web: www.youreko.com
Mobile: +44 (0)7809 296438
Skype: patrick_verdon

This entire communication is sent on behalf of
Youreko Ltd and is strictly confidential to and
for the sole use of the intended addressee.

Registered in England - 7448349

--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat