RE: Segmentation Fault - too many proxy balancers

[Paul Beckett <paul_beckett@xxxxxxxxxxx>]

Jeff,

Thanks for your help with this. I'm not on the very latest mod_security version (few months old). I'll try upgrading it, and following up with mod_security community if that doesn't resolve it. Thanks for the tip about valgrind, I'll look into that (this is new territory for me).

Thanks again,

Paul

---

Date: Sun, 27 Jul 2014 08:27:04 -0400
From: trawick@xxxxxxxxx
To: users@xxxxxxxxxxxxxxxx
Subject: Re: Segmentation Fault - too many proxy balancers

On Sat, Jul 26, 2014 at 1:37 PM, Paul Beckett <paul_beckett@xxxxxxxxxxx> wrote:

Just realised I should probably have typed where in gdb after that, which additionally yields:

(gdb) where

#0  apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:97

#1  0x00007ffe08a90e83 in sec_audit_logger (msr=0x7ffdac006328) at msc_logging.c:579

#2  0x00007ffe08a966f7 in modsecurity_process_phase_logging (msr=0x7ffdac006328, phase=<value optimized out>) at modsecurity.c:689

#3  modsecurity_process_phase (msr=0x7ffdac006328, phase=<value optimized out>) at modsecurity.c:795

#4  0x00007ffe08a6b68b in hook_log_transaction (r=0x7ffdac004980) at mod_security2.c:1202

So the crash is during request logging, in mod_security, as it passes a bad value to apr_global_mutex_lock().

Check if you have the latest version of mod_security and/or if those folks have seen this crash before.

Increasing some type of resource (e.g., number of proxy balancers) can push things around in memory and perhaps a latent bug is now exposed.

Sometimes running httpd under a memory debugger like valgrind can catch bad use of memory at an earlier stage.  Consider that if you can reproduce this in a test environment.

 

#5  0x00000000004343f0 in ap_run_log_transaction (r=0x7ffdac004980) at protocol.c:1788

#6  0x000000000044571f in eor_bucket_cleanup (data="" optimized out>) at eor_bucket.c:35

#7  0x00007ffe10040c6e in run_cleanups (pool=0x7ffdac004908) at memory/unix/apr_pools.c:2352

#8  apr_pool_destroy (pool=0x7ffdac004908) at memory/unix/apr_pools.c:804

#9  0x00007ffe0cb3b9b9 in ssl_io_filter_output (f=0x7ffdb4001240, bb=0x7ffd5c003d98) at ssl_engine_io.c:1659

#10 0x00007ffe0cb3c0f5 in ssl_io_filter_coalesce (f=0x7ffdb4001218, bb=0x7ffd5c003d98) at ssl_engine_io.c:1558

#11 0x000000000045fa5d in ap_process_request_after_handler (r=0x7ffdac004980) at http_request.c:256

#12 0x000000000045d040 in ap_process_http_async_connection (c=0x7ffdb4000cc0) at http_core.c:143

#13 ap_process_http_connection (c=0x7ffdb4000cc0) at http_core.c:228

#14 0x0000000000454b30 in ap_run_process_connection (c=0x7ffdb4000cc0) at connection.c:41

#15 0x00007ffe0eda17e1 in process_socket (thd=0x2047658, dummy=<value optimized out>) at event.c:970

#16 worker_thread (thd=0x2047658, dummy=<value optimized out>) at event.c:1815

#17 0x00007ffe0f9cf851 in start_thread () from /lib64/libpthread.so.0

#18 0x00007ffe0f71d90d in clone () from /lib64/libc.so.6

From: paul_beckett@xxxxxxxxxxx
To: users@xxxxxxxxxxxxxxxx
Date: Sat, 26 Jul 2014 18:35:03 +0100
Subject: RE: Segmentation Fault - too many proxy balancers

Jeff,

Thanks for the response. I'm not very familiar with the process of creating a backtrace - I've not had to debug apache at this level before (trying to follow info at http://httpd.apache.org/dev/debugging.html#backtrace ). 

Initially I created a backtrace using pstack, but this was about 1 second prior to the segmentation fault, so I guess this is probably not useful (included at end of email just in case)

I then enabled core dumps, and used gdb to analyse it, output below. Not knowing what to expect from the backtrace, is this info what you were expecting when you requested I post the backtrace?

Thanks again,

Paul

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/local/apache/bin/httpd...done.

[New Thread 10172]

[New Thread 10174]

[New Thread 10175]

[New Thread 10180]

[New Thread 10148]

[New Thread 10186]

[New Thread 10184]

[New Thread 10181]

[New Thread 10200]

[New Thread 10209]

[New Thread 10173]

[New Thread 10178]

[New Thread 10155]

[New Thread 10167]

[New Thread 10171]

[New Thread 10208]

[New Thread 10212]

[New Thread 10159]

[New Thread 10182]

[New Thread 10158]

[New Thread 10149]

[New Thread 10185]

[New Thread 10183]

[New Thread 10203]

[New Thread 10160]

[New Thread 10162]

[New Thread 10213]

[New Thread 10164]

[New Thread 10188]

[New Thread 10150]

[New Thread 10187]

[New Thread 10177]

[New Thread 10189]

[New Thread 10168]

[New Thread 10154]

[New Thread 10215]

[New Thread 10165]

[New Thread 10156]

[New Thread 10214]

[New Thread 10163]

[New Thread 10198]

[New Thread 10170]

[New Thread 10190]

[New Thread 10179]

[New Thread 10217]

[New Thread 10216]

[New Thread 10202]

[New Thread 10196]

[New Thread 10191]

[New Thread 10205]

[New Thread 10195]

[New Thread 10192]

[New Thread 10207]

[New Thread 10204]

[New Thread 10251]

[New Thread 10201]

[New Thread 10194]

[New Thread 10199]

[New Thread 10193]

[New Thread 10161]

[New Thread 10219]

[New Thread 10153]

[New Thread 10249]

[New Thread 10166]

[New Thread 10218]

[New Thread 10197]

[New Thread 10222]

[New Thread 10247]

[New Thread 10157]

[New Thread 10220]

[New Thread 10245]

[New Thread 10221]

[New Thread 10224]

[New Thread 10232]

[New Thread 10243]

[New Thread 10235]

[New Thread 10250]

[New Thread 10241]

[New Thread 10230]

[New Thread 10206]

[New Thread 10233]

[New Thread 10237]

[New Thread 10248]

[New Thread 10228]

[New Thread 10239]

[New Thread 10211]

[New Thread 10231]

[New Thread 10252]

[New Thread 10238]

[New Thread 10236]

[New Thread 10226]

[New Thread 10210]

[New Thread 10229]

[New Thread 10240]

[New Thread 10234]

[New Thread 10227]

[New Thread 10223]

[New Thread 10242]

[New Thread 10152]

[New Thread 10225]

[New Thread 10176]

[New Thread 10169]

[New Thread 10246]

[New Thread 10244]

Missing separate debuginfo for /usr/local/apache/lib/libaprutil-1.so.0

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/37/3e4c965981a9edf122d4ae60548613d9ae2336

Missing separate debuginfo for /usr/local/apache/lib/libapr-1.so.0

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8d/960ff682eb3d2312c241d82c9afbedd2577d2b

Missing separate debuginfo for /usr/local/apache/lib/libexpat.so.0

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/29/61a04fd4c6f84d8dac7ea2c2e3db1d4a6f3b59

Missing separate debuginfo for /usr/local/apache/modules/mod_shib_24.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/aa/ebeac19976d4611374e125ecd112bf1ade2011

Missing separate debuginfo for /usr/local/shibsp/lib/libshibsp-lite.so.6

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f1/b9a9fe4772f4c2750129e7872d3b63f870e83c

Missing separate debuginfo for /usr/local/apache/modules/mod_mpm_event.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6d/324996bc4cd28c75ad3e9106c2893491e3efcf

Missing separate debuginfo for /usr/local/apache/modules/mod_unixd.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/57d49fd65d7561bf944f507da9d7dad2e513a9

Missing separate debuginfo for /usr/local/apache/modules/mod_dir.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bd/7d85b4dd733009451d18b3414c1fd76d6b8a4d

Missing separate debuginfo for /usr/local/apache/modules/mod_log_config.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/5d/620abb41bd1b466156aac66e3a2200714f6f53

Missing separate debuginfo for /usr/local/apache/modules/mod_mime.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/eb/b8f8bf64db622502d39c625e3a9d88a0ae1e82

Missing separate debuginfo for /usr/local/apache/modules/mod_mime_magic.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6f/f24d8f77831d3206d541e587578af86ab2a519

Missing separate debuginfo for /usr/local/apache/modules/mod_authn_core.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8f/bec33c7e6f4a3cacdece68f32fff862d0f7b30

Missing separate debuginfo for /usr/local/apache/modules/mod_authz_host.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9c/ad185c4b505641acc2217c6bcb8532974dd489

Missing separate debuginfo for /usr/local/apache/modules/mod_authz_user.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4b/07ae5e53f6c9d42455589a56cc52dc21e022a9

Missing separate debuginfo for /usr/local/apache/modules/mod_authz_core.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fa/d69123abc0b4e1e6bb7338e7754c5dae94dd70

Missing separate debuginfo for /usr/local/apache/modules/mod_access_compat.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a3/2700ac62cd51565e1563c6852c6ff996573576

Missing separate debuginfo for /usr/local/apache/modules/mod_reqtimeout.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/d6/9d6cf839eceb72c55235064dee28fc9e0d01b0

Missing separate debuginfo for /usr/local/apache/modules/mod_filter.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/17/56caae1845d0adc08fb069208ef3101ae718da

Missing separate debuginfo for /usr/local/apache/modules/mod_env.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4e/36fea0939efac30d11bf5cec681fc7d171ee45

Missing separate debuginfo for /usr/local/apache/modules/mod_include.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b6/380ff50a323d5711c065a70f17052d3ba2da29

Missing separate debuginfo for /usr/local/apache/modules/mod_headers.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0e/10b837e15f98e4ecdb7dc785dcc0381f57e8f6

Missing separate debuginfo for /usr/local/apache/modules/mod_setenvif.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/dc/08448a6e9c167ceb128981105847af62175e12

Missing separate debuginfo for /usr/local/apache/modules/mod_ssl.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ab/2e79aba60c6bc949754958f36ca9899d7e0c0f

Missing separate debuginfo for /usr/local/apache/modules/mod_socache_shmcb.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6a/a31378be3a2741f0dcc17abc413ca31300cd7d

Missing separate debuginfo for /usr/local/apache/modules/mod_rewrite.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/75/1362a102f672d1ded8dd8ea09e597a152b8f3e

Missing separate debuginfo for /usr/local/apache/modules/mod_status.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bd/1f9b60389f1e4ee2ed1edf918ba03b3de1b077

Missing separate debuginfo for /usr/local/apache/modules/mod_info.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c8/6f92dfae31509c068568a4913567db8f5e083c

Missing separate debuginfo for /usr/local/apache/modules/mod_cache.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/75/d7a334e208aeb7b07b7b81514b863b40117be1

Missing separate debuginfo for /usr/local/apache/modules/mod_cache_disk.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bf/2989446a71cdffd99631f310ebba25c0f24e0b

Missing separate debuginfo for /usr/local/apache/modules/mod_deflate.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/58/cf80ecc07f6ce5eddf682298c9793d7755d753

Missing separate debuginfo for /usr/local/apache/modules/mod_slotmem_shm.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e4/c513d6ecd62be795c6cd32f4928f82a36d7e64

Missing separate debuginfo for /usr/local/apache/modules/mod_proxy.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/17/1a1a7350d5d2f111046d77f4bc075e335695d1

Missing separate debuginfo for /usr/local/apache/modules/mod_proxy_connect.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9d/9b005ecad7b6c16f5b46363efb5bf5e595a026

Missing separate debuginfo for /usr/local/apache/modules/mod_proxy_http.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/98/6860aef9605a6db743ee2ae027dc36e8a77f26

Missing separate debuginfo for /usr/local/apache/modules/mod_proxy_balancer.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e6/62c31160985d77b5019b87ec10ef0b2d07f618

Missing separate debuginfo for /usr/local/apache/modules/mod_lbmethod_byrequests.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/58/3071a430fd84abd068fe2aafb8d9074b0e36c1

Missing separate debuginfo for /usr/local/apache/modules/mod_unique_id.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/65/a9a5a3b4aeb09caded6e5fac915c3a3533ef53

Missing separate debuginfo for /usr/local/apache/modules/mod_security2.so

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/16/38ce80438b7cabfa13da67aaeb976d29a5f9ac

Missing separate debuginfo for

Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/08/f634a1d22deff00461d50a7699dacdc97657bf

Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib64/libpcre.so.0

Reading symbols from /usr/local/apache/lib/libaprutil-1.so.0...done.

Loaded symbols for /usr/local/apache/lib/libaprutil-1.so.0

Reading symbols from /lib64/libexpat.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libexpat.so.1

Reading symbols from /usr/local/apache/lib/libapr-1.so.0...done.

Loaded symbols for /usr/local/apache/lib/libapr-1.so.0

Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/librt.so.1

Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypt.so.1

Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib64/libpthread.so.0

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /usr/local/apache/lib/libexpat.so.0...done.

Loaded symbols for /usr/local/apache/lib/libexpat.so.0

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libfreebl3.so

Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libresolv.so.2

Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /usr/local/apache/modules/mod_shib_24.so...done.

Loaded symbols for /usr/local/apache/modules/mod_shib_24.so

Reading symbols from /usr/local/shibsp/lib/libshibsp-lite.so.6...done.

Loaded symbols for /usr/local/shibsp/lib/libshibsp-lite.so.6

Reading symbols from /usr/lib64/libxmltooling-lite.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libxmltooling-lite.so.6

Reading symbols from /usr/lib64/libxerces-c-3.1.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libxerces-c-3.1.so

Reading symbols from /usr/lib64/liblog4shib.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/liblog4shib.so.1

Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnsl.so.1

Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libstdc++.so.6

Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libgcc_s.so.1

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

Reading symbols from /lib64/libnss_vas4.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_vas4.so.2

Reading symbols from /opt/quest/lib64/libvtsmartcache.so.1...(no debugging symbols found)...done.

Loaded symbols for /opt/quest/lib64/libvtsmartcache.so.1

Reading symbols from /opt/quest/lib64/libvtutil.so.1...(no debugging symbols found)...done.

Loaded symbols for /opt/quest/lib64/libvtutil.so.1

Reading symbols from /opt/quest/lib64/libvtcacheipc.so.1...(no debugging symbols found)...done.

Loaded symbols for /opt/quest/lib64/libvtcacheipc.so.1

Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_dns.so.2

Reading symbols from /usr/lib64/gconv/UTF-16.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/gconv/UTF-16.so

Reading symbols from /usr/local/apache/modules/mod_mpm_event.so...done.

Loaded symbols for /usr/local/apache/modules/mod_mpm_event.so

Reading symbols from /usr/local/apache/modules/mod_unixd.so...done.

Loaded symbols for /usr/local/apache/modules/mod_unixd.so

Reading symbols from /usr/local/apache/modules/mod_dir.so...done.

Loaded symbols for /usr/local/apache/modules/mod_dir.so

Reading symbols from /usr/local/apache/modules/mod_log_config.so...done.

Loaded symbols for /usr/local/apache/modules/mod_log_config.so

Reading symbols from /usr/local/apache/modules/mod_mime.so...done.

Loaded symbols for /usr/local/apache/modules/mod_mime.so

Reading symbols from /usr/local/apache/modules/mod_mime_magic.so...done.

Loaded symbols for /usr/local/apache/modules/mod_mime_magic.so

Reading symbols from /usr/local/apache/modules/mod_authn_core.so...done.

Loaded symbols for /usr/local/apache/modules/mod_authn_core.so

Reading symbols from /usr/local/apache/modules/mod_authz_host.so...done.

Loaded symbols for /usr/local/apache/modules/mod_authz_host.so

Reading symbols from /usr/local/apache/modules/mod_authz_user.so...done.

Loaded symbols for /usr/local/apache/modules/mod_authz_user.so

Reading symbols from /usr/local/apache/modules/mod_authz_core.so...done.

Loaded symbols for /usr/local/apache/modules/mod_authz_core.so

Reading symbols from /usr/local/apache/modules/mod_access_compat.so...done.

Loaded symbols for /usr/local/apache/modules/mod_access_compat.so

Reading symbols from /usr/local/apache/modules/mod_reqtimeout.so...done.

Loaded symbols for /usr/local/apache/modules/mod_reqtimeout.so

Reading symbols from /usr/local/apache/modules/mod_filter.so...done.

Loaded symbols for /usr/local/apache/modules/mod_filter.so

Reading symbols from /usr/local/apache/modules/mod_env.so...done.

Loaded symbols for /usr/local/apache/modules/mod_env.so

Reading symbols from /usr/local/apache/modules/mod_include.so...done.

Loaded symbols for /usr/local/apache/modules/mod_include.so

Reading symbols from /usr/local/apache/modules/mod_headers.so...done.

Loaded symbols for /usr/local/apache/modules/mod_headers.so

Reading symbols from /usr/local/apache/modules/mod_setenvif.so...done.

Loaded symbols for /usr/local/apache/modules/mod_setenvif.so

Reading symbols from /usr/local/apache/modules/mod_ssl.so...done.

Loaded symbols for /usr/local/apache/modules/mod_ssl.so

Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssl.so.10

Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libcrypto.so.10

Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libz.so.1

Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libgssapi_krb5.so.2

Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkrb5.so.3

Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcom_err.so.2

Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.

Loaded symbols for /lib64/libk5crypto.so.3

Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkrb5support.so.0

Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkeyutils.so.1

Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libselinux.so.1

Reading symbols from /usr/local/apache/modules/mod_socache_shmcb.so...done.

Loaded symbols for /usr/local/apache/modules/mod_socache_shmcb.so

Reading symbols from /usr/local/apache/modules/mod_rewrite.so...done.

Loaded symbols for /usr/local/apache/modules/mod_rewrite.so

Reading symbols from /usr/local/apache/modules/mod_status.so...done.

Loaded symbols for /usr/local/apache/modules/mod_status.so

Reading symbols from /usr/local/apache/modules/mod_info.so...done.

Loaded symbols for /usr/local/apache/modules/mod_info.so

Reading symbols from /usr/local/apache/modules/mod_cache.so...done.

Loaded symbols for /usr/local/apache/modules/mod_cache.so

Reading symbols from /usr/local/apache/modules/mod_cache_disk.so...done.

Loaded symbols for /usr/local/apache/modules/mod_cache_disk.so

Reading symbols from /usr/local/apache/modules/mod_deflate.so...done.

Loaded symbols for /usr/local/apache/modules/mod_deflate.so

Reading symbols from /usr/local/apache/modules/mod_slotmem_shm.so...done.

Loaded symbols for /usr/local/apache/modules/mod_slotmem_shm.so

Reading symbols from /usr/local/apache/modules/mod_proxy.so...done.

Loaded symbols for /usr/local/apache/modules/mod_proxy.so

Reading symbols from /usr/local/apache/modules/mod_proxy_connect.so...done.

Loaded symbols for /usr/local/apache/modules/mod_proxy_connect.so

Reading symbols from /usr/local/apache/modules/mod_proxy_http.so...done.

Loaded symbols for /usr/local/apache/modules/mod_proxy_http.so

Reading symbols from /usr/local/apache/modules/mod_proxy_balancer.so...done.

Loaded symbols for /usr/local/apache/modules/mod_proxy_balancer.so

Reading symbols from /usr/local/apache/modules/mod_lbmethod_byrequests.so...done.

Loaded symbols for /usr/local/apache/modules/mod_lbmethod_byrequests.so

Reading symbols from /usr/lib64/libxml2.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libxml2.so

Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/liblua-5.1.so

Reading symbols from /usr/local/apache/modules/mod_unique_id.so...done.

Loaded symbols for /usr/local/apache/modules/mod_unique_id.so

Reading symbols from /usr/local/apache/modules/mod_security2.so...done.

Loaded symbols for /usr/local/apache/modules/mod_security2.so

Core was generated by `/usr/local/apache/bin/httpd -k start -D UEA_TEST_SERVER'.

Program terminated with signal 11, Segmentation fault.

#0  apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:97

97          if (mutex->thread_mutex) {

Missing separate debuginfos, use: debuginfo-install expat-2.0.1-11.el6_2.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.2.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.7-3.el6.x86_64 liblog4shib1-1.0.6-1.1.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libstdc++-4.4.7-3.el6.x86_64 libxerces-c-3_1-3.1.1-2.4.el6.x86_64 libxml2-devel-2.7.6-12.el6_4.1.x86_64 libxmltooling6-1.5.3-1.1.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 openssl-1.0.0-27.el6_4.2.x86_64 pcre-7.8-6.el6.x86_64 zlib-1.2.3-29.el6.x86_64

Thread 104 (Thread 0x7fe663364700 (LWP 64967)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66531db79 in xmltooling::ReloadableXMLFile::reload_fn(void*) () from /usr/lib64/libxmltooling-lite.so.6

#2  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#3  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 103 (Thread 0x7fe662963700 (LWP 64968)):

#0  0x00007fe66ca767bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe665326bc4 in xmltooling::CondWaitImpl::timedwait(xmltooling::Mutex*, int) () from /usr/lib64/libxmltooling-lite.so.6

#2  0x00007fe66562b26d in (anonymous namespace)::SSCache::cleanup_fn(void*) () from /usr/local/shibsp/lib/libshibsp-lite.so.6

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 102 (Thread 0x7fe661561700 (LWP 64970)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 101 (Thread 0x7fe660b60700 (LWP 64971)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 100 (Thread 0x7fe653fff700 (LWP 64972)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 99 (Thread 0x7fe6535fe700 (LWP 64973)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 98 (Thread 0x7fe652bfd700 (LWP 64974)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 97 (Thread 0x7fe6521fc700 (LWP 64975)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 96 (Thread 0x7fe6517fb700 (LWP 64976)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 95 (Thread 0x7fe650dfa700 (LWP 64977)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 94 (Thread 0x7fe6503f9700 (LWP 64978)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 93 (Thread 0x7fe64f9f8700 (LWP 64979)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 92 (Thread 0x7fe64eff7700 (LWP 64980)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 91 (Thread 0x7fe64e5f6700 (LWP 64981)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 90 (Thread 0x7fe64dbf5700 (LWP 64982)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 89 (Thread 0x7fe64d1f4700 (LWP 64983)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 88 (Thread 0x7fe64c7f3700 (LWP 64984)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 87 (Thread 0x7fe64bdf2700 (LWP 64985)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 86 (Thread 0x7fe64b3f1700 (LWP 64986)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 85 (Thread 0x7fe64a9f0700 (LWP 64987)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 84 (Thread 0x7fe649fef700 (LWP 64988)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 83 (Thread 0x7fe6495ee700 (LWP 64989)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 82 (Thread 0x7fe648bed700 (LWP 64990)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 81 (Thread 0x7fe6481ec700 (LWP 64991)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 80 (Thread 0x7fe6477eb700 (LWP 64992)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 79 (Thread 0x7fe646dea700 (LWP 64993)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 78 (Thread 0x7fe6463e9700 (LWP 64994)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 77 (Thread 0x7fe6459e8700 (LWP 64995)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 76 (Thread 0x7fe644fe7700 (LWP 64996)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 75 (Thread 0x7fe6445e6700 (LWP 64997)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 74 (Thread 0x7fe643be5700 (LWP 64998)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 73 (Thread 0x7fe6431e4700 (LWP 64999)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 72 (Thread 0x7fe6427e3700 (LWP 65000)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 71 (Thread 0x7fe641de2700 (LWP 65001)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 70 (Thread 0x7fe6413e1700 (LWP 65002)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 69 (Thread 0x7fe6409e0700 (LWP 65003)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 68 (Thread 0x7fe63ffdf700 (LWP 65004)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 67 (Thread 0x7fe63f5de700 (LWP 65005)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 66 (Thread 0x7fe63ebdd700 (LWP 65006)):

#0  0x00007fe66c7b7253 in poll () from /lib64/libc.so.6

#1  0x00007fe66d0ed367 in apr_wait_for_io_or_timeout () from /usr/local/apache/lib/libapr-1.so.0

#2  0x00007fe66d0e780a in apr_socket_recv () from /usr/local/apache/lib/libapr-1.so.0

#3  0x00007fe66d52c255 in socket_bucket_read () from /usr/local/apache/lib/libaprutil-1.so.0

#4  0x000000000044689e in ap_core_input_filter ()

#5  0x00007fe66a819e14 in reqtimeout_filter () from /usr/local/apache/modules/mod_reqtimeout.so

#6  0x00007fe669be01b9 in bio_filter_in_read () from /usr/local/apache/modules/mod_ssl.so

#7  0x00007fe669681419 in BIO_read () from /usr/lib64/libcrypto.so.10

#8  0x00007fe669992bb8 in ssl3_read_n () from /usr/lib64/libssl.so.10

#9  0x00007fe66999327d in ssl3_read_bytes () from /usr/lib64/libssl.so.10

#10 0x00007fe66998f700 in ?? () from /usr/lib64/libssl.so.10

#11 0x00007fe669bdf3fe in ssl_io_input_read () from /usr/local/apache/modules/mod_ssl.so

#12 0x00007fe669bdfac8 in ssl_io_filter_input () from /usr/local/apache/modules/mod_ssl.so

#13 0x0000000000435ec2 in ap_rgetline_core ()

#14 0x0000000000436cee in ap_read_request ()

#15 0x000000000045cfc9 in ap_process_http_connection ()

#16 0x0000000000454b30 in ap_run_process_connection ()

#17 0x00007fe66be447e1 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#18 0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#19 0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 65 (Thread 0x7fe63e1dc700 (LWP 65007)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 64 (Thread 0x7fe63d7db700 (LWP 65008)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7fe63cdda700 (LWP 65009)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7fe63c3d9700 (LWP 65010)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7fe63b9d8700 (LWP 65011)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7fe63afd7700 (LWP 65012)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 59 (Thread 0x7fe63a5d6700 (LWP 65013)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7fe639bd5700 (LWP 65014)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7fe6391d4700 (LWP 65015)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7fe6387d3700 (LWP 65016)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7fe637dd2700 (LWP 65017)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7fe6373d1700 (LWP 65018)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7fe6369d0700 (LWP 65019)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7fe635fcf700 (LWP 65020)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7fe6355ce700 (LWP 65021)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7fe634bcd700 (LWP 65022)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7fe6341cc700 (LWP 65023)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7fe6337cb700 (LWP 65024)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7fe632dca700 (LWP 65025)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7fe6323c9700 (LWP 65026)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7fe6319c8700 (LWP 65027)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7fe630fc7700 (LWP 65028)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7fe6305c6700 (LWP 65029)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7fe62fbc5700 (LWP 65030)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7fe62f1c4700 (LWP 65031)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7fe62e7c3700 (LWP 65032)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7fe62ddc2700 (LWP 65033)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7fe62d3c1700 (LWP 65034)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7fe62c9c0700 (LWP 65035)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7fe62bfbf700 (LWP 65036)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7fe62b5be700 (LWP 65037)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7fe62abbd700 (LWP 65038)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7fe62a1bc700 (LWP 65039)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7fe6297bb700 (LWP 65040)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7fe628dba700 (LWP 65041)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7fe6283b9700 (LWP 65042)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fe6279b8700 (LWP 65043)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fe626fb7700 (LWP 65044)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fe6265b6700 (LWP 65045)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fe625bb5700 (LWP 65046)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fe6251b4700 (LWP 65047)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fe6247b3700 (LWP 65048)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fe623db2700 (LWP 65049)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fe6233b1700 (LWP 65050)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fe6229b0700 (LWP 65051)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fe621faf700 (LWP 65052)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fe6215ae700 (LWP 65053)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fe620bad700 (LWP 65054)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fe6201ac700 (LWP 65055)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fe61f7ab700 (LWP 65056)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fe61edaa700 (LWP 65057)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fe61e3a9700 (LWP 65058)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fe61d9a8700 (LWP 65059)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fe61cfa7700 (LWP 65060)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fe61c5a6700 (LWP 65061)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fe61bba5700 (LWP 65062)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fe61b1a4700 (LWP 65063)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fe61a7a3700 (LWP 65064)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fe619da2700 (LWP 65065)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fe6193a1700 (LWP 65066)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fe6189a0700 (LWP 65067)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fe617f9f700 (LWP 65068)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fe61759e700 (LWP 65069)):

#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so

#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fe616b9d700 (LWP 65070)):

#0  0x00007fe66c7c0f03 in epoll_wait () from /lib64/libc.so.6

#1  0x00007fe66d0ea271 in impl_pollset_poll () from /usr/local/apache/lib/libapr-1.so.0

#2  0x00007fe66be4333b in listener_thread () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fe66db80700 (LWP 64966)):

#0  0x00007fe66ca7954d in read () from /lib64/libpthread.so.0

#1  0x00000000004577b7 in ap_mpm_podx_check ()

#2  0x00007fe66be44eec in child_main () from /usr/local/apache/modules/mod_mpm_event.so

#3  0x00007fe66be45978 in make_child () from /usr/local/apache/modules/mod_mpm_event.so

#4  0x00007fe66be46554 in event_run () from /usr/local/apache/modules/mod_mpm_event.so

#5  0x000000000043089e in ap_run_mpm ()

#6  0x000000000042ab84 in main ()

---

Date: Sat, 26 Jul 2014 08:28:07 -0400
From: trawick@xxxxxxxxx
To: users@xxxxxxxxxxxxxxxx
Subject: Re: Segmentation Fault - too many proxy balancers

On Fri, Jul 25, 2014 at 2:00 PM, Paul Beckett <paul_beckett@xxxxxxxxxxx> wrote:

My apache server has started segmentation faulting all the time (seems to log a segmentation fault every few requests to the apache error log):

[Fri Jul 25 06:25:42.046752 2014] [core:notice] [pid 11226:tid 140006078953216] AH00052: child pid 11715 exit signal Segmentation fault (11)

Can you post a backtrace from the crash?

 

This appears to be due to the number of proxy balancers I have configured (problem isn't related to any one specific proxy balancer, adding / removing any of the proxy balancers causes the problem to appear/disappear). I'm using Apache HTTPD as a reverse proxy for a lot of load-balanced (by apache httpd) application servers. My googling so far hasn't found any specific limit on the number of proxy, or how I can increase this. 

I am running Apache HTTPD 2.4.9, built from source on RHEL6.

I would be very grateful if anyone can shed more light on this, and assuming I'm right about a limit: point my in the right direction as to how I can increase this.

Thanks,
Paul

--

Born in Roswell... married an alien...
http://emptyhammock.com/

http://edjective.org/

--

Born in Roswell... married an alien...
http://emptyhammock.com/

http://edjective.org/