ØHello everyone
I have a problem with apache.
I run Solaris 9, on a SunFire 210.
The apache server is acting as a front web server for a websphere 5.1 Portal server, I run version 2.0.52 of apache
The apache server is configured and compiled from scratch with the following:
./configure --prefix=/usr/local/apache_prod --with-mpm=worker --enable-info --enable-so --enable-rewrite
My problem is that, the httpd process takes 100% cpu, sometimes it can go for a couple of hours with 0.8 or 0.14% cpu and acts fine, but then suddenly it grows, and it stays and eats up all the cpu, and I have to restart.
I have tried to pstop the process, but when I do prun again, it grows back to eat the cpu
I have used prstat -m -L -s cpu -p <pid>
and I can see that often its the same threads that has the problem, they stay on the cpu for hours, and they have very large ICX (involunteray context switches) values.
When I use truss to check out what its doing, i cant see any action from the threads that have high ICX
Can anyone give me some help in what I can do to investigate theese threads/lwp further, I know about pstack, but I am not able to read hex.
I attach the hex from the two lwp's/threads with high ICX just in case :-)
----------------- lwp# 25 / thread# 25 --------------------
ff309a5c apr_pool_cleanup_kill (1f4edf8, 20bab8, ffffffff, 66, 2f, 68) + 20
00031f2c ap_http_header_filter (1fa778, 1f4edf8, 1e03f1, 1f505f0, 68, 31ae4) + 448
0005ef8c ap_pass_brigade (1fa778, 1f4edf8, fd37b4e0, 1f505f0, 9a7e8, 0) + 44
00061560 ap_content_length_filter (1, 1f4edf8, ff375870, 1f505f0, 0, 614c4) + 9c
0005ef8c ap_pass_brigade (1fa760, 1f4edf8, 1f4edf8, 1f4edfc, 1f4edfc, 1cac28) + 44
0005ef8c ap_pass_brigade (1fad30, 1f4ee18, 1fa760, 20bab8, 1fad30, 1f4ee1c) + 44
00061c04 ap_rflush (1f9af0, ffffffff, 1f9af0, 0, 0, 0) + 44
feda80e0 cb_write_body (fd37bba4, 20bab8, ffffffff, 0, 0, 0) + f0
fedbef4c writeResponseBody (fd37b910, 20bab8, ffffffff, 66, 2f, 68) + 2c
fedc7794 esiResponsePassThru (fd37b910, 1c3828, 1e03f1, 48, 68, ff0a7c14) + e4
fedc7310 esiResponseWrite (1c3850, 1, 1, 0, 786d6c, 73000000) + 28
fedc3c64 esiHandleRequest (fd37b910, 1, ffffffff, 7efefeff, 81010100, ff00) + f4
fedb22f8 websphereHandleRequest (fd37bba4, 1fa830, f4240, f4240, 74, 72) + 350
feda7560 as_handler (1f9af0, 1faa88, 8, 0, 99694, 155f88) + 150
00051348 ap_run_handler (1f9af0, 0, 3b, 0, 155a88, 155b08) + 3c
00051908 ap_invoke_handler (1028e8, 84000, 1f9af0, 0, 0, fffffffe) + 78
00034230 ap_process_request (1f9af0, 16, 4, 1f9af0, 9a2e8, 16) + 84
0002f6b4 ap_process_http_connection (1c6c30, 1c6b40, 1c6b40, 56, 9a2e8, 156458) + f4
0005cb3c ap_run_process_connection (1c6c30, 1c6b40, 1c6b40, 56, 1c6c28, 1cab18) + 3c
0004dfe8 process_socket (1c6b08, 1c6b40, 1, 16, 1cab18, e75f0) + 68
0004e6c0 worker_thread (e7918, 99400, 0, 0, 0, 0) + 15c
ff30487c dummy_worker (e7918, 0, 0, 0, 0, 0) + c
fef85854 _lwp_start (0, 0, 0, 0, 0, 0)
-------------------------------------------------------------------------------------------------
----------------- lwp# 13 / thread# 13 --------------------
ff309a5c apr_pool_cleanup_kill (1f4d5e0, 1061720, ffffffff, 66, 2f, 68) + 20
00031f2c ap_http_header_filter (1e86d0, 1f4d5e0, ff4919, 1f4e5e8, 68, 31ae4) + 448
0005ef8c ap_pass_brigade (1e86d0, 1f4d5e0, fdf7b4e0, 1f4e5e8, 9a7e8, 0) + 44
00061560 ap_content_length_filter (1, 1f4d5e0, ff375870, 1f4e5e8, 1edbb48, 614c4) + 9c
0005ef8c ap_pass_brigade (1e86b8, 1f4d5e0, 1f4d5e0, 1f4d5e4, 1f4d5e4, 1d9ba8) + 44
0005ef8c ap_pass_brigade (1e8d58, 1f4d600, 1e86b8, 1061720, 1e8d58, 1f4d604) + 44
00061c04 ap_rflush (1e7a48, ffffffff, 1e7a48, 0, 0, 0) + 44
feda80e0 cb_write_body (fdf7bba4, 1061720, ffffffff, 0, 0, 0) + f0
fedbef4c writeResponseBody (fdf7b910, 1061720, ffffffff, 66, 2f, 68) + 2c
fedc7794 esiResponsePassThru (fdf7b910, 5c89c8, ff4919, 48, 68, ff0a7c14) + e4
fedc7310 esiResponseWrite (6db380, 1, 1, 0, da8aa5, 73000000) + 28
fedc3c64 esiHandleRequest (fdf7b910, 1, ffffffff, 7efefeff, 81010100, ff00) + f4
fedb22f8 websphereHandleRequest (fdf7bba4, 1e8788, f4240, f4240, 74, 72) + 350
feda7560 as_handler (1e7a48, 1e8ab0, 8, 0, 99694, 155f88) + 150
00051348 ap_run_handler (1e7a48, 0, 3b, 0, 155a88, 155b08) + 3c
00051908 ap_invoke_handler (1028e8, 84000, 1e7a48, 0, 0, fffffffe) + 78
00034230 ap_process_request (1e7a48, a, 4, 1e7a48, 9a2e8, a) + 84
0002f6b4 ap_process_http_connection (1d2c60, 1d2b70, 1d2b70, 4a, 9a2e8, 156458) + f4
0005cb3c ap_run_process_connection (1d2c60, 1d2b70, 1d2b70, 4a, 1d2c58, 1d9a98) + 3c
0004dfe8 process_socket (1d2b38, 1d2b70, 1, a, 1d9a98, e75f0) + 68
0004e6c0 worker_thread (e7798, 99400, 0, 0, 0, 0) + 15c
ff30487c dummy_worker (e7798, 0, 0, 0, 0, 0) + c
fef85854 _lwp_start (0, 0, 0, 0, 0, 0)
-------------------------------------------------------------------------------------------------------------
Also, here is a snapshot of the following command:
prstat -m -L -s cpu -p 427
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
427 www1 73 0.0 0.1 0.0 0.0 0.0 0.0 27 0 255 0 0 httpd/13
427 www1 59 0.0 0.0 0.0 0.0 0.0 0.0 41 0 166 0 0 httpd/25
427 www1 0.1 0.1 0.0 0.0 0.0 40 60 0.0 31 2 74 0 httpd/3
427 www1 0.0 0.0 0.0 0.0 0.0 25 75 0.0 4 0 5 0 httpd/17
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 1 0 2 0 httpd/28
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/27
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/26
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/24
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/23
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 httpd/22
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/21
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/20
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/19
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/18
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/16
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/15
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/14
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/12
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/11
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/10
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/9
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/8
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/7
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/6
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/5
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 httpd/4
427 www1 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 httpd/1
----------------------------------------------------------------------------------------------------------------------
Thank you
Mvh
Kjell Grindalen
This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal.
|