Re: Apache 2.4 DoS? (Apache Users)

Hi Ranier Caravan,

Looks like this problem didn't go away.

To get symbols I had to first:

sudo debuginfo-install httpd

Then

sudo gdb -p PID

Several hundred apache forks are present so I selected random one....

(gdb) where

#0 0x00007f247f302f0d in poll () at ../sysdeps/unix/syscall-template.S:81

#1 0x00007f247fa1e94b in poll (__timeout=30000, __nfds=1, __fds=0x7ffd5cd92b00) at /usr/include/bits/poll2.h:46

#2 apr_wait_for_io_or_timeout (f=f@entry=0x0, s=s@entry=0x55a9964b7610, for_read=for_read@entry=1) at support/unix/waitio.c:51

#3 0x00007f247fa1877a in apr_socket_recv (sock=sock@entry=0x55a9964b7610,

buf=buf@entry=0x55a996541b08 "2\r\nh\n\r\n2\r\nh\n\r\n2\r\nh\n\r\n2\r\nh\n\r\ny: Express\r\nCache-Control: no-store, no-cache, no-transform, must-revalidate, max-age=0\r\nAccess-Control-Allow-Origin: *\r\nVary: Origin\r\nContent-Type: application/_javascript_;"..., len=len@entry=0x7ffd5cd92be0) at network_io/unix/sendrecv.c:87

#4 0x00007f2480457281 in socket_bucket_read (a=0x55a99653dc48, str=0x7ffd5cd92bd8, len=0x7ffd5cd92be0, block=<optimized out>) at buckets/apr_buckets_socket.c:36

#5 0x00007f2480455756 in apr_brigade_split_line (bbOut=bbOut@entry=0x55a9964d8ed8, bbIn=0x55a9964b7f28, block=block@entry=APR_BLOCK_READ, maxbytes=maxbytes@entry=8192) at buckets/apr_brigade.c:319

#6 0x000055a993422d5c in ap_core_input_filter (f=0x55a9964b7e28, b=0x55a9964d8ed8, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core_filters.c:147

#7 0x00007f247a134c4e in logio_in_filter (f=<optimized out>, bb=0x55a9964d8ed8, mode=<optimized out>, block=<optimized out>, readbytes=<optimized out>) at mod_logio.c:140

#8 0x000055a99343e2bd in ap_http_filter (f=<optimized out>, b=<optimized out>, mode=<optimized out>, block=<optimized out>, readbytes=8192) at http_filters.c:515

#9 0x00007f2474eed058 in ap_proxy_http_process_response (p=p@entry=0x55a9965018e8, r=r@entry=0x55a996501960, backend_ptr=backend_ptr@entry=0x7ffd5cd97058, server_portstr=server_portstr@entry=0x7ffd5cd97100 "", conf=0x55a994beaef0,

conf=0x55a994beaef0, worker=<optimized out>) at mod_proxy_http.c:1708

#10 0x00007f2474eef7f9 in proxy_http_handler (r=<optimized out>, worker=<optimized out>, conf=0x55a994beaef0, url="" "http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming",

proxyname=0x0, proxyport=0) at mod_proxy_http.c:2038

#11 0x00007f247673edc4 in proxy_run_scheme_handler (r=r@entry=0x55a996501960, worker=0x55a994beb1b0, conf=conf@entry=0x55a994beaef0,

url="" "http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming", proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:2746

#12 0x00007f247673fcad in proxy_handler (r=0x55a996501960) at mod_proxy.c:1125

#13 0x000055a993427990 in ap_run_handler (r=r@entry=0x55a996501960) at config.c:169

#14 0x000055a993427ed9 in ap_invoke_handler (r=r@entry=0x55a996501960) at config.c:439

#15 0x000055a99343ca8a in ap_process_async_request (r=r@entry=0x55a996501960) at http_request.c:328

#16 0x000055a99343cd64 in ap_process_request (r=r@entry=0x55a996501960) at http_request.c:363

#17 0x000055a993438f62 in ap_process_http_sync_connection (c=0x55a9964a9090) at http_core.c:190

#18 ap_process_http_connection (c=0x55a9964a9090) at http_core.c:231

#19 0x000055a993430fc0 in ap_run_process_connection (c=c@entry=0x55a9964a9090) at connection.c:41

#20 0x000055a9934313a8 in ap_process_connection (c=c@entry=0x55a9964a9090, csd=<optimized out>) at connection.c:202

#21 0x00007f24769547af in child_main (child_num_arg=child_num_arg@entry=6) at prefork.c:707

#22 0x00007f24769549f5 in make_child (s=0x55a994af9e30, slot=6) at prefork.c:810

#23 0x00007f247695568e in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:912

#24 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1100

#25 0x000055a99340bffe in ap_run_mpm (pconf=pconf@entry=0x55a994ab5138, plog=0x55a994ae2358, s=0x55a994af9e30) at mpm_common.c:96

#26 0x000055a993404d76 in main (argc=2, argv=0x7ffd5cd97878) at main.c:783

bt full for 10 and 11:

proxyname=0x0, proxyport=0) at mod_proxy_http.c:2038

locurl = 0x55a9964d8490 "/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming"

status = <optimized out>

server_portstr = "\000vM\226\251U\000\000\350\030P\226\251U\000\000\000q\331\\\375\177\000\000\000\000\000\000\000\000\000"

scheme = <optimized out>

proxy_function = 0x7f2474ef0bfa "HTTP"

u = <optimized out>

backend = 0x55a9964b5600

is_ssl = 0

c = 0x55a9964a9090

retry = 0

p = <optimized out>

uri = 0x55a9964d83b0

#11 0x00007f247673edc4 in proxy_run_scheme_handler (r=r@entry=0x55a996501960, worker=0x55a994beb1b0, conf=conf@entry=0x55a994beaef0,

url="" "http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming", proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:2746

pHook = <optimized out>

n = 3

rv = -1

I am beyond my knowledge limit at this point.

The server nike.pbtech contains an R application running on internal server. It's reached through this Apace proxy.

I have a few questions.

1) How do we tell the apache fork's hung hung?

2) How can I check all forks at once? Attaching to parent does not work.

3) When trying other Apache forks I get hung gdb at line "attaching to process." Any way to avoid?

Thanks,

Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit

Physiology and Biophysics

Weill Cornell Medicine

E: doug@xxxxxxxxxxxxxxx
O: 212-746-6305
F: 212-746-8690

On Mon, Nov 13, 2017 at 7:04 AM, Rainer Canavan <rainer.canavan@xxxxxxxxxxxx> wrote:

On Fri, Nov 10, 2017 at 6:41 PM, Douglas Duckworth
<dod2014@xxxxxxxxxxxxxxx> wrote:
> Hi
>
> I am running old PHP under Apache httpd-2.4.

[...]
> Though, ever few weeks, we see sudden increase in workers who never seem to
> retire:
>
> [Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584] AH00161:
> server reached MaxRequestWorkers setting, consider raising the
> MaxRequestWorkers setting
>
> user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
> 257

If the php locks up while processing your request, no logs will be
written. You may be running
into a bug where circular, unresolvable dependencies for a lock
prevent the processes from
completing their requests. To check what's going on, install gdb, the
debug info for your php
and httpd and find the .gdbinfo that came with the httpd and php
version you're using. Then
attach gdb to any of the hanging processes (gdb `which httpd` PID),
source both .gdbinit files,
do a "zbacktrace" and a "bt full", and repeat for some other hanging
processes. Depending
on the type of lock, you may be able to identify the first process
that has acquired that lock
that all others are waiting for, and the php code and / or php module
that causes it.

rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx