Jeff, The cwmp handler has no functions like 'request_handler' and 'release_lock'. The internal redirect from cwmp handler goes to weblogic apache plugin which has to tunnel the request to a weblogic server cluster. Thanks, ~Abhi ----- Original Message ----- From: Jeff Trawick <trawick@xxxxxxxxx> To: users@xxxxxxxxxxxxxxxx Cc: Sent: Thursday, July 19, 2012 4:49 PM Subject: Re: Apache httpd 2.2.21 Segmentation faults On Thu, Jul 19, 2012 at 5:18 AM, Abhi Auradkar <aurabhi@xxxxxxxxx> wrote: > Hi, > > I tried attaching the debugger to apache running as a single instance(httpd -X) and head also turned on memcheck and accesscheck(check -all) with librtc. > I saw a core dump and it had this stack. Is something going wrong in internal_redirect()? > > (dbx) where > current thread: t@3 > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_core.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_core.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/connection.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/connection.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" > dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" > [1] __rtc_trap_handler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfee96e24 > [2] 0x0(0x1205e0, 0x0, 0x0, 0x0, 0x0, 0xb), at 0x0 This is the point of failure. release_lock() is branching to zero. > [3] release_lock(0x169050, 0x1cc8b0, 0x0, 0x189af0, 0x5, 0x1010101), at 0xfe1504a8 > [4] request_handler(0x1539e0, 0x11d588, 0x1539e0, 0x0, 0x0, 0xfffeb748), at 0xfe15dfa8 request_handler and release_lock are third-party code. (same module as the cwmp function seen earlier in the stack?) This needs to be debugged from the perspective of that third-party code -- what was release_lock() trying to do when it branched to location zero, and why wasn't a valid function pointer available. > [5] ap_run_handler(0x1539e0, 0x1cd7a8, 0x1717ca, 0xfef27378, 0xfee96c9c, 0xfee93dec), at 0x50314 > [6] ap_invoke_handler(r = 0x1539e0), at 0x50fa4 > [7] ap_internal_redirect_handler(new_uri = 0x12bdf0 "/cwmp-allow/cwmpWeb/CPEMgt", r = 0x152ed0), at 0x7e34c > =>[8] cwmp_throttle_handler(r = 0x152ed0) (optimized), at 0xfe697c2c (line ~2403) in "mod_cwmp.c" > [9] ap_run_handler(0x152ed0, 0x159ef0, 0x152ed0, 0x10b098, 0x153068, 0x153070), at 0x50314 > [10] ap_invoke_handler(r = 0x152ed0), at 0x50fa4 > [11] ap_process_request(r = 0x152ed0), at 0x7d6dc > [12] ap_process_http_connection(c = 0x10b098), at 0x78e98 > [13] ap_run_process_connection(0x10b098, 0x168268, 0x168268, 0x0, 0x1cddf0, 0xf79d0), at 0x5cba4 > [14] ap_process_connection(c = 0x10b098, csd = 0x168268), at 0x5d270 > [15] process_socket(p = 0x110fe8, sock = 0x168268, my_child_num = 0, my_thread_num = 0, bucket_alloc = 0xf79d0), at 0x91e8c > [16] worker_thread(thd = 0x110e40, dummy = 0xe8690), at 0x92a68 > [17] dummy_worker(opaque = 0x110e40), line 142 in "thread.c" > (dbx) threads > > > > ----- Original Message ----- > From: Jeff Trawick <trawick@xxxxxxxxx> > To: users@xxxxxxxxxxxxxxxx; Abhi Auradkar <aurabhi@xxxxxxxxx> > Cc: > Sent: Wednesday, June 6, 2012 1:25 AM > Subject: Re: Apache httpd 2.2.21 Segmentation faults > > On Mon, Jun 4, 2012 at 2:27 AM, Abhi Auradkar <aurabhi@xxxxxxxxx> wrote: >> Hi All, >> >> >> I am running a apache 2.2.21 on Solaris Sparc. I see the following >> segmantation faults every now and then. >> Any pointers to whats wrong will be really helpful. > > Please get the stack of this thread, which is probably the one that crashed: > > t@68 a l@68 dummy_worker() LWP suspended in in <can't get PC>() > > As is also visible with pflags sometimes, Solaris can misidentify the > main thread of the process as the crasher. That is blocked in read() > during steady state and cannot crash. > >> >> Thanks. >> >> t@1 (l@1) terminated by signal SEGV (Segmentation Fault) >> 0xff03e3ac: _read+0x000c: bcc,pt %icc,_read+0x20 ! 0xff03e3c0 >> (dbx) where >> current thread: t@1 >> =>[1] _read(0x4, 0xffbff84b, 0x1, 0x0, 0x0, 0x0), at 0xff03e3ac >> [2] read(0x8, 0xffbff84b, 0x1, 0x0, 0x1cf4, 0x8), at 0xff02ca1c >> [3] ap_mpm_pod_check(0xd6d70, 0x6b854, 0xa0000, 0xff232a00, 0xfff7, 0x0), >> at 0x6f408 >> [4] child_main(0x5, 0xffbff8b4, 0x135d98, 0x1, 0x9f0ec, 0x1b0870), at >> 0x6cbbc >> [5] perform_idle_server_maintenance(0x0, 0x9f000, 0xafb70, 0x2, 0x5, 0x0), >> at 0x6d6ac >> [6] server_main_loop(0x0, 0x0, 0x5, 0x9f0b0, 0x0, 0xffbffa78), at 0x6dd10 >> [7] ap_mpm_run(0x20, 0xa07d8, 0x9f0b8, 0x0, 0x20, 0x60), at 0x6e000 >> [8] main(0x9e800, 0x4b8d0, 0x9ebb4, 0x7ddd0, 0x9d8e8, 0x9e800), at 0x29eac >> (dbx) threads >> o> t@1 a l@1 ?() signal SIGSEGV in _read() >> t@3 a l@3 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@4 a l@4 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@5 a l@5 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@6 a l@6 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@7 a l@7 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@8 a l@8 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@9 a l@9 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@10 a l@10 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@11 a l@11 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@12 a l@12 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@13 a l@13 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@14 a l@14 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@15 a l@15 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@16 a l@16 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@17 a l@17 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@18 a l@18 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@19 a l@19 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@20 a l@20 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@21 a l@21 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@22 a l@22 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@23 a l@23 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@24 a l@24 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@25 a l@25 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@26 a l@26 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@27 a l@27 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@28 a l@28 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@29 a l@29 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@30 a l@30 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@31 a l@31 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@32 a l@32 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@33 a l@33 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@34 a l@34 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@35 a l@35 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@36 a l@36 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@37 a l@37 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@38 a l@38 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@39 a l@39 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@40 a l@40 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@41 a l@41 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@42 a l@42 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@43 a l@43 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@44 a l@44 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@45 a l@45 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@46 a l@46 dummy_worker() LWP suspended in _stat64() >> t@47 a l@47 dummy_worker() LWP suspended in __lwp_park() >> t@48 a l@48 dummy_worker() LWP suspended in __pollsys() >> t@49 a l@49 dummy_worker() LWP suspended in __pollsys() >> t@50 a l@50 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@51 a l@51 dummy_worker() LWP suspended in strstr() >> t@52 a l@52 dummy_worker() LWP suspended in __pollsys() >> t@53 a l@53 dummy_worker() LWP suspended in __pollsys() >> t@54 a l@54 dummy_worker() LWP suspended in __pollsys() >> t@55 a l@55 dummy_worker() sleep on 0xff0b5a60 in __lwp_park() >> t@56 a l@56 dummy_worker() LWP suspended in _lstat64() >> t@57 a l@57 dummy_worker() LWP suspended in __pollsys() >> t@58 a l@58 dummy_worker() LWP suspended in _brk_unlocked() >> t@59 a l@59 dummy_worker() LWP suspended in __pollsys() >> t@60 a l@60 dummy_worker() LWP suspended in _lstat64() >> t@61 a l@61 dummy_worker() LWP suspended in __pollsys() >> t@62 a l@62 dummy_worker() LWP suspended in __pollsys() >> t@63 a l@63 dummy_worker() LWP suspended in _lstat64() >> t@64 a l@64 dummy_worker() LWP suspended in __lwp_park() >> t@65 a l@65 dummy_worker() LWP suspended in __pollsys() >> t@66 a l@66 dummy_worker() sleep on 0x135e30 in __lwp_park() >> t@67 a l@67 dummy_worker() LWP suspended in __lwp_unpark() >> t@68 a l@68 dummy_worker() LWP suspended in in <can't get PC>() >> t@69 a l@69 dummy_worker() LWP suspended in ___nanosleep() >> (dbx) >> >> >> Apache Version Info >> ============= >> >> /usr/local/apache2/bin/httpd -v >> Server version: Apache/2.2.21 (Unix) >> Server built: Jan 19 2012 09:52:37 >> bash-3.00# /usr/local/apache2/bin/httpd -V >> Server version: Apache/2.2.21 (Unix) >> Server built: Jan 19 2012 09:52:37 >> Server's Module Magic Number: 20051115:30 >> Server loaded: APR 1.4.5, APR-Util 1.3.12 >> Compiled using: APR 1.4.5, APR-Util 1.3.12 >> Architecture: 32-bit >> Server MPM: Worker >> threaded: yes (fixed thread count) >> forked: yes (variable process count) >> Server compiled with.... >> -D APACHE_MPM_DIR="server/mpm/worker" >> -D APR_HAS_SENDFILE >> -D APR_HAS_MMAP >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >> -D APR_USE_PROC_PTHREAD_SERIALIZE >> -D APR_USE_PTHREAD_SERIALIZE >> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >> -D APR_HAS_OTHER_CHILD >> -D AP_HAVE_RELIABLE_PIPED_LOGS >> -D DYNAMIC_MODULE_LIMIT=128 >> -D HTTPD_ROOT="/usr/local/apache2" >> -D SUEXEC_BIN="/usr/local/apache2/bin/suexec" >> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >> -D DEFAULT_ERRORLOG="logs/error_log" >> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >> -D SERVER_CONFIG_FILE="conf/httpd.conf" >> >> > > > > -- > Born in Roswell... married an alien... > http://emptyhammock.com/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx > -- Born in Roswell... married an alien... http://emptyhammock.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|