----- Original Message ----- From: Jeff Trawick <trawick@xxxxxxxxx> To: users@xxxxxxxxxxxxxxxx Cc: Sent: Thursday, July 19, 2012 5:15 PM Subject: Re: Apache httpd 2.2.21 Segmentation faults On Thu, Jul 19, 2012 at 7:32 AM, Abhi Auradkar <aurabhi@xxxxxxxxx> wrote: > Jeff, > > The cwmp handler has no functions like 'request_handler' and 'release_lock'. > The internal redirect from cwmp handler goes to weblogic apache plugin which has to tunnel the request to a weblogic server cluster. Interesting... I suggest trying the latest WLS plugin release (11.1.1.6.0) and, if that doesn't resolve the problem, contact Oracle support. > > > > Thanks, > ~Abhi > > ----- Original Message ----- > > From: Jeff Trawick <trawick@xxxxxxxxx> > To: users@xxxxxxxxxxxxxxxx > Cc: > Sent: Thursday, July 19, 2012 4:49 PM > Subject: Re: Apache httpd 2.2.21 Segmentation faults > > On Thu, Jul 19, 2012 at 5:18 AM, Abhi Auradkar <aurabhi@xxxxxxxxx> wrote: >> Hi, >> >> I tried attaching the debugger to apache running as a single instance(httpd -X) and head also turned on memcheck and accesscheck(check -all) with librtc. >> I saw a core dump and it had this stack. Is something going wrong in internal_redirect()? >> >> (dbx) where >> current thread: t@3 >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/config.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_request.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_core.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/modules/http/http_core.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/connection.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/connection.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" >> dbx: warning: can't find file "/data/build_area/xml-data/build-dir/HDM-APACHEHTTPD2218-APACHE2221DEBUG/server/mpm/worker/worker.c" >> [1] __rtc_trap_handler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfee96e24 >> [2] 0x0(0x1205e0, 0x0, 0x0, 0x0, 0x0, 0xb), at 0x0 > > This is the point of failure. release_lock() is branching to zero. > >> [3] release_lock(0x169050, 0x1cc8b0, 0x0, 0x189af0, 0x5, 0x1010101), at 0xfe1504a8 >> [4] request_handler(0x1539e0, 0x11d588, 0x1539e0, 0x0, 0x0, 0xfffeb748), at 0xfe15dfa8 > > request_handler and release_lock are third-party code. (same module > as the cwmp function seen earlier in the stack?) > > This needs to be debugged from the perspective of that third-party > code -- what was release_lock() trying to do when it branched to > location zero, and why wasn't a valid function pointer available. > >> [5] ap_run_handler(0x1539e0, 0x1cd7a8, 0x1717ca, 0xfef27378, 0xfee96c9c, 0xfee93dec), at 0x50314 >> [6] ap_invoke_handler(r = 0x1539e0), at 0x50fa4 >> [7] ap_internal_redirect_handler(new_uri = 0x12bdf0 "/cwmp-allow/cwmpWeb/CPEMgt", r = 0x152ed0), at 0x7e34c >> =>[8] cwmp_throttle_handler(r = 0x152ed0) (optimized), at 0xfe697c2c (line ~2403) in "mod_cwmp.c" >> [9] ap_run_handler(0x152ed0, 0x159ef0, 0x152ed0, 0x10b098, 0x153068, 0x153070), at 0x50314 >> [10] ap_invoke_handler(r = 0x152ed0), at 0x50fa4 >> [11] ap_process_request(r = 0x152ed0), at 0x7d6dc >> [12] ap_process_http_connection(c = 0x10b098), at 0x78e98 >> [13] ap_run_process_connection(0x10b098, 0x168268, 0x168268, 0x0, 0x1cddf0, 0xf79d0), at 0x5cba4 >> [14] ap_process_connection(c = 0x10b098, csd = 0x168268), at 0x5d270 >> [15] process_socket(p = 0x110fe8, sock = 0x168268, my_child_num = 0, my_thread_num = 0, bucket_alloc = 0xf79d0), at 0x91e8c >> [16] worker_thread(thd = 0x110e40, dummy = 0xe8690), at 0x92a68 >> [17] dummy_worker(opaque = 0x110e40), line 142 in "thread.c" >> (dbx) threads >> >> >> >> ----- Original Message ----- >> From: Jeff Trawick <trawick@xxxxxxxxx> >> To: users@xxxxxxxxxxxxxxxx; Abhi Auradkar <aurabhi@xxxxxxxxx> >> Cc: >> Sent: Wednesday, June 6, 2012 1:25 AM >> Subject: Re: Apache httpd 2.2.21 Segmentation faults >> >> On Mon, Jun 4, 2012 at 2:27 AM, Abhi Auradkar <aurabhi@xxxxxxxxx> wrote: >>> Hi All, >>> >>> >>> I am running a apache 2.2.21 on Solaris Sparc. I see the following >>> segmantation faults every now and then. >>> Any pointers to whats wrong will be really helpful. >> >> Please get the stack of this thread, which is probably the one that crashed: >> >> t@68 a l@68 dummy_worker() LWP suspended in in <can't get PC>() >> >> As is also visible with pflags sometimes, Solaris can misidentify the >> main thread of the process as the crasher. That is blocked in read() >> during steady state and cannot crash. >> >>> >>> Thanks. >>> >>> t@1 (l@1) terminated by signal SEGV (Segmentation Fault) >>> 0xff03e3ac: _read+0x000c: bcc,pt %icc,_read+0x20 ! 0xff03e3c0 >>> (dbx) where >>> current thread: t@1 >>> =>[1] _read(0x4, 0xffbff84b, 0x1, 0x0, 0x0, 0x0), at 0xff03e3ac >>> [2] read(0x8, 0xffbff84b, 0x1, 0x0, 0x1cf4, 0x8), at 0xff02ca1c >>> [3] ap_mpm_pod_check(0xd6d70, 0x6b854, 0xa0000, 0xff232a00, 0xfff7, 0x0), >>> at 0x6f408 >>> [4] child_main(0x5, 0xffbff8b4, 0x135d98, 0x1, 0x9f0ec, 0x1b0870), at >>> 0x6cbbc >>> [5] perform_idle_server_maintenance(0x0, 0x9f000, 0xafb70, 0x2, 0x5, 0x0), >>> at 0x6d6ac >>> [6] server_main_loop(0x0, 0x0, 0x5, 0x9f0b0, 0x0, 0xffbffa78), at 0x6dd10 >>> [7] ap_mpm_run(0x20, 0xa07d8, 0x9f0b8, 0x0, 0x20, 0x60), at 0x6e000 >>> [8] main(0x9e800, 0x4b8d0, 0x9ebb4, 0x7ddd0, 0x9d8e8, 0x9e800), at 0x29eac >>> (dbx) threads >>> o> t@1 a l@1 ?() signal SIGSEGV in _read() >>> t@3 a l@3 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@4 a l@4 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@5 a l@5 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@6 a l@6 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@7 a l@7 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@8 a l@8 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@9 a l@9 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@10 a l@10 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@11 a l@11 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@12 a l@12 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@13 a l@13 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@14 a l@14 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@15 a l@15 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@16 a l@16 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@17 a l@17 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@18 a l@18 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@19 a l@19 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@20 a l@20 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@21 a l@21 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@22 a l@22 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@23 a l@23 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@24 a l@24 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@25 a l@25 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@26 a l@26 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@27 a l@27 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@28 a l@28 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@29 a l@29 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@30 a l@30 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@31 a l@31 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@32 a l@32 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@33 a l@33 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@34 a l@34 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@35 a l@35 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@36 a l@36 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@37 a l@37 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@38 a l@38 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@39 a l@39 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@40 a l@40 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@41 a l@41 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@42 a l@42 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@43 a l@43 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@44 a l@44 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@45 a l@45 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@46 a l@46 dummy_worker() LWP suspended in _stat64() >>> t@47 a l@47 dummy_worker() LWP suspended in __lwp_park() >>> t@48 a l@48 dummy_worker() LWP suspended in __pollsys() >>> t@49 a l@49 dummy_worker() LWP suspended in __pollsys() >>> t@50 a l@50 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@51 a l@51 dummy_worker() LWP suspended in strstr() >>> t@52 a l@52 dummy_worker() LWP suspended in __pollsys() >>> t@53 a l@53 dummy_worker() LWP suspended in __pollsys() >>> t@54 a l@54 dummy_worker() LWP suspended in __pollsys() >>> t@55 a l@55 dummy_worker() sleep on 0xff0b5a60 in __lwp_park() >>> t@56 a l@56 dummy_worker() LWP suspended in _lstat64() >>> t@57 a l@57 dummy_worker() LWP suspended in __pollsys() >>> t@58 a l@58 dummy_worker() LWP suspended in _brk_unlocked() >>> t@59 a l@59 dummy_worker() LWP suspended in __pollsys() >>> t@60 a l@60 dummy_worker() LWP suspended in _lstat64() >>> t@61 a l@61 dummy_worker() LWP suspended in __pollsys() >>> t@62 a l@62 dummy_worker() LWP suspended in __pollsys() >>> t@63 a l@63 dummy_worker() LWP suspended in _lstat64() >>> t@64 a l@64 dummy_worker() LWP suspended in __lwp_park() >>> t@65 a l@65 dummy_worker() LWP suspended in __pollsys() >>> t@66 a l@66 dummy_worker() sleep on 0x135e30 in __lwp_park() >>> t@67 a l@67 dummy_worker() LWP suspended in __lwp_unpark() >>> t@68 a l@68 dummy_worker() LWP suspended in in <can't get PC>() >>> t@69 a l@69 dummy_worker() LWP suspended in ___nanosleep() >>> (dbx) >>> >>> >>> Apache Version Info >>> ============= >>> >>> /usr/local/apache2/bin/httpd -v >>> Server version: Apache/2.2.21 (Unix) >>> Server built: Jan 19 2012 09:52:37 >>> bash-3.00# /usr/local/apache2/bin/httpd -V >>> Server version: Apache/2.2.21 (Unix) >>> Server built: Jan 19 2012 09:52:37 >>> Server's Module Magic Number: 20051115:30 >>> Server loaded: APR 1.4.5, APR-Util 1.3.12 >>> Compiled using: APR 1.4.5, APR-Util 1.3.12 >>> Architecture: 32-bit >>> Server MPM: Worker >>> threaded: yes (fixed thread count) >>> forked: yes (variable process count) >>> Server compiled with.... >>> -D APACHE_MPM_DIR="server/mpm/worker" >>> -D APR_HAS_SENDFILE >>> -D APR_HAS_MMAP >>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>> -D APR_USE_PROC_PTHREAD_SERIALIZE >>> -D APR_USE_PTHREAD_SERIALIZE >>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>> -D APR_HAS_OTHER_CHILD >>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>> -D DYNAMIC_MODULE_LIMIT=128 >>> -D HTTPD_ROOT="/usr/local/apache2" >>> -D SUEXEC_BIN="/usr/local/apache2/bin/suexec" >>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>> -D DEFAULT_ERRORLOG="logs/error_log" >>> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>> >>> >> >> >> >> -- >> Born in Roswell... married an alien... >> http://emptyhammock.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx >> > > > > -- > Born in Roswell... married an alien... > http://emptyhammock.com/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx > -- Born in Roswell... married an alien... http://emptyhammock.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|