Is your test server the same architecture with same library
versions and os version as the production servers?
Have you tried putting a simulated load on the test server to
see if it falls over under the same load as the production box has?
Have you used strace on any of the production apache processes
to get a better look at what’s going on when a process segfaults?
-Tony
From:
rats1027-segfault@xxxxxxxxxxx [mailto:rats1027-segfault@xxxxxxxxxxx]
Sent: Friday, March 27, 2009 11:27 AM
To: users@xxxxxxxxxxxxxxxx
Subject: Re: [users@httpd] need help on "Program terminated with
signal 11, Segmentation fault"
guys, still we are facing the same problem.
we recompiled new php source(to latest version), upgraded httpd to 2.2.11,
removed .htaccess rules but no use.
can you guys suggest me any other ways of debugging pls? actually i am not a
system administration guy. I am a php developer with little knowledge on httpd.
But we have system administration group who are working on this problem.
just pls check out again the segfault statement that we are getting:
segfault at 00007fff84114ff8 rip 00002aaab1bd47ac rsp 00007fff84115000 error 6
that bold line is repeating many times randomly on our log file and always it is same on
specific server. we have 6 load balanced servers and all are giving the same error randomly.
address locations are different from each server but for a specific server it is same always
and every day.
you guys think any kind of attack thats happening over our systems? actually we do not accept
much input from users other than a keyword(for search) which can be an unicode and we have some
POST requests which are purely browser generated data(no user data here).
im again pasting our gdb backtraces of two different core dumps for reference:
> #0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
> No symbol table info available.
> #1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
> No symbol table info
available.
> #2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from /usr/sbin/httpd
> No symbol table info available.
> #3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
> No symbol table info available.
> #4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
> No symbol table info available.
> #5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
> No symbol table info available.
> #6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
> No symbol table info available.
above thing is happening many times say 10 to 15 times a minute(actualy
depends on the traffic at that time)
> #0 0x00002aaab1c46688 in ZEND_FETCH_DIM_RW_SPEC_VAR_UNUSED_HANDLER (
> execute_data=0x5555714ea6c8)
> at
/usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h:13204
> opline = (zend_op *) 0x5555714e8798
> free_op1 = {var = 0x2aaaac1454fd}
> #1 0x00005555714e86b8 in ?? ()
> No symbol table info available.
> #2 0x00002aaaac145afb in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
> No symbol table info available.
> #3 0x000055555556a27b in suck_in_APR () from /usr/sbin/httpd
> No
> symbol table info available.
> #4 0x000055555556adf6 in main () from /usr/sbin/httpd
> No symbol table info available.
above thing is happening in very lesser frequency compared to the above.
--thank you so much guys, rats.
From: "rats1027-segfault@xxxxxxxxxxx"
<rats1027-segfault@xxxxxxxxxxx>
To: users@xxxxxxxxxxxxxxxx
Sent: Wednesday, 25 March, 2009 9:09:54 PM
Subject: Re: [users@httpd] need help on "Program terminated with
signal 11, Segmentation fault"
hi frank, thank you so much for the reply. we are using
'prefork' and our server is architecture is : x86-64.
you can see the loaded modules for apache2hanlder below
core
prefork
http_core
mod_so
mod_authz_host
mod_authz_user
mod_authz_owner
mod_authz_groupfile
mod_authz_dbm
mod_authz_default
mod_include
mod_log_config
mod_env
mod_mime_magic
mod_expires
mod_headers
mod_setenvif
mod_mime
mod_status
mod_autoindex
mod_asis
mod_negotiation
mod_dir
mod_alias
mod_rewrite
mod_php5
--rats
From: Frank Gingras
<francois.gingras@xxxxxxxxx>
To: users@xxxxxxxxxxxxxxxx
Sent: Wednesday, 25 March, 2009 8:53:28 PM
Subject: Re: [users@httpd] need help on "Program terminated with
signal 11, Segmentation fault"
Are you using worker, by any chance?
Frank
rats1027-segfault@xxxxxxxxxxx
wrote:
> Hi Guys,
>
> We are have a problem with apache segfault on our production server.
Please read bellow for description.
>
> Its a web application written in php5 and implemented most of the oop
concepts and lot of regular expressions, curl, mcrypt, simplexml, mssql,
exceptions and user defined error handlers. When we run this app on our test
server, we had no problems, but when we moved it on to the production(here we
used have big amount of traffic), initially we saw no problems from our end
user testing but from system log, we saw lots of 'segfaults' and thus requests
were being dropped(difference in traffic).
>
> Weird thing is that, on the same apache httpd, there is an another
application running successfully which is having lesser oop concepts but with
same libraries. We are running these two applications with virtual host
concept. We see 'segfaults' only when the traffic is very high on the first
application.
>
> We have upgraded our php module from 5.2.6 to 5.2.9 but with no result. We
have the core dump of the apache below:
>
> #0 0x000055555557dfee in ap_merge_per_dir_configs () from
/usr/sbin/httpd
> No symbol table info available.
> #1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
> No symbol table info available.
> #2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from
/usr/sbin/httpd
> No symbol table info available.
> #3 0x0000555555578b42 in ap_run_map_to_storage () from
/usr/sbin/httpd
> No symbol table info available.
> #4 0x0000555555579cbc in ap_process_request_internal () from
/usr/sbin/httpd
> No symbol table info available.
> #5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
> No symbol table info available.
> #6 0x0000555555588900 in ap_register_input_filter () from
/usr/sbin/httpd
> No symbol table info available.
> #7 0x0000555555584a92 in ap_run_process_connection () from
/usr/sbin/httpd
> No symbol table info available.
> #8 0x000055555558f27b in ap_graceful_stop_signalled () from
/usr/sbin/httpd
> No symbol table info available.
> #9 0x000055555558f50a in ap_graceful_stop_signalled () from
/usr/sbin/httpd
> No symbol table info available.
> #10 0x000055555558fd7b in ap_mpm_run () from /usr/sbin/httpd
> No symbol table info available.
> #11 0x000055555556ade4 in main () from /usr/sbin/httpd
> No symbol table info available.
>
> We have tried modifying most of the curl implementation but with no use.
Also now we have no clues of the origin of the bug. Any kind of help regarding
this is mostly appreciated.
>
> THANK YOU
> --rats
>
>
>
> Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html>
for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
Add more friends to your messenger and enjoy! Invite them now.
Add more friends to your messenger and enjoy! Invite
them now.
|
