[users@httpd] Apache 2.0.52 eats all cpu, and have high ICX

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Title: Apache 2.0.52 eats all cpu, and have high ICX

ØHello everyone

I have a problem with apache.

I run Solaris 9, on a SunFire 210.
The apache server is acting as a front web server for a websphere 5.1 Portal server, I run version 2.0.52 of apache

The apache server is configured and compiled from scratch with the following:
./configure --prefix=/usr/local/apache_prod --with-mpm=worker --enable-info --enable-so --enable-rewrite

My problem is that, the httpd process takes 100% cpu, sometimes it can go for a couple of hours with 0.8 or 0.14% cpu and acts fine, but then suddenly it grows, and it stays and eats up all the cpu, and I have to restart.
I have tried to pstop the process, but when I do prun again, it grows back to eat the cpu

I have used prstat -m -L -s cpu -p <pid>
and I can see that often its the same threads that has the problem, they stay on the cpu for hours, and they have very large ICX  (involunteray context switches) values.

When I use truss to check out what its doing, i cant see any action from the threads that have high ICX

Can anyone give me some help in what I can do to investigate theese threads/lwp further, I know about pstack, but I am not able to read hex.

I attach the hex from the two lwp's/threads with high ICX just in case :-)
-----------------  lwp# 25 / thread# 25  --------------------
 ff309a5c apr_pool_cleanup_kill (1f4edf8, 20bab8, ffffffff, 66, 2f, 68) + 20
 00031f2c ap_http_header_filter (1fa778, 1f4edf8, 1e03f1, 1f505f0, 68, 31ae4) + 448
 0005ef8c ap_pass_brigade (1fa778, 1f4edf8, fd37b4e0, 1f505f0, 9a7e8, 0) + 44
 00061560 ap_content_length_filter (1, 1f4edf8, ff375870, 1f505f0, 0, 614c4) + 9c
 0005ef8c ap_pass_brigade (1fa760, 1f4edf8, 1f4edf8, 1f4edfc, 1f4edfc, 1cac28) + 44
 0005ef8c ap_pass_brigade (1fad30, 1f4ee18, 1fa760, 20bab8, 1fad30, 1f4ee1c) + 44
 00061c04 ap_rflush (1f9af0, ffffffff, 1f9af0, 0, 0, 0) + 44
 feda80e0 cb_write_body (fd37bba4, 20bab8, ffffffff, 0, 0, 0) + f0
 fedbef4c writeResponseBody (fd37b910, 20bab8, ffffffff, 66, 2f, 68) + 2c
 fedc7794 esiResponsePassThru (fd37b910, 1c3828, 1e03f1, 48, 68, ff0a7c14) + e4
 fedc7310 esiResponseWrite (1c3850, 1, 1, 0, 786d6c, 73000000) + 28
 fedc3c64 esiHandleRequest (fd37b910, 1, ffffffff, 7efefeff, 81010100, ff00) + f4
 fedb22f8 websphereHandleRequest (fd37bba4, 1fa830, f4240, f4240, 74, 72) + 350
 feda7560 as_handler (1f9af0, 1faa88, 8, 0, 99694, 155f88) + 150
 00051348 ap_run_handler (1f9af0, 0, 3b, 0, 155a88, 155b08) + 3c
 00051908 ap_invoke_handler (1028e8, 84000, 1f9af0, 0, 0, fffffffe) + 78
 00034230 ap_process_request (1f9af0, 16, 4, 1f9af0, 9a2e8, 16) + 84
 0002f6b4 ap_process_http_connection (1c6c30, 1c6b40, 1c6b40, 56, 9a2e8, 156458) + f4
 0005cb3c ap_run_process_connection (1c6c30, 1c6b40, 1c6b40, 56, 1c6c28, 1cab18) + 3c
 0004dfe8 process_socket (1c6b08, 1c6b40, 1, 16, 1cab18, e75f0) + 68
 0004e6c0 worker_thread (e7918, 99400, 0, 0, 0, 0) + 15c
 ff30487c dummy_worker (e7918, 0, 0, 0, 0, 0) + c
 fef85854 _lwp_start (0, 0, 0, 0, 0, 0)
-------------------------------------------------------------------------------------------------

-----------------  lwp# 13 / thread# 13  --------------------
 ff309a5c apr_pool_cleanup_kill (1f4d5e0, 1061720, ffffffff, 66, 2f, 68) + 20
 00031f2c ap_http_header_filter (1e86d0, 1f4d5e0, ff4919, 1f4e5e8, 68, 31ae4) + 448
 0005ef8c ap_pass_brigade (1e86d0, 1f4d5e0, fdf7b4e0, 1f4e5e8, 9a7e8, 0) + 44
 00061560 ap_content_length_filter (1, 1f4d5e0, ff375870, 1f4e5e8, 1edbb48, 614c4) + 9c
 0005ef8c ap_pass_brigade (1e86b8, 1f4d5e0, 1f4d5e0, 1f4d5e4, 1f4d5e4, 1d9ba8) + 44
 0005ef8c ap_pass_brigade (1e8d58, 1f4d600, 1e86b8, 1061720, 1e8d58, 1f4d604) + 44
 00061c04 ap_rflush (1e7a48, ffffffff, 1e7a48, 0, 0, 0) + 44
 feda80e0 cb_write_body (fdf7bba4, 1061720, ffffffff, 0, 0, 0) + f0
 fedbef4c writeResponseBody (fdf7b910, 1061720, ffffffff, 66, 2f, 68) + 2c
 fedc7794 esiResponsePassThru (fdf7b910, 5c89c8, ff4919, 48, 68, ff0a7c14) + e4
 fedc7310 esiResponseWrite (6db380, 1, 1, 0, da8aa5, 73000000) + 28
 fedc3c64 esiHandleRequest (fdf7b910, 1, ffffffff, 7efefeff, 81010100, ff00) + f4
 fedb22f8 websphereHandleRequest (fdf7bba4, 1e8788, f4240, f4240, 74, 72) + 350
 feda7560 as_handler (1e7a48, 1e8ab0, 8, 0, 99694, 155f88) + 150
 00051348 ap_run_handler (1e7a48, 0, 3b, 0, 155a88, 155b08) + 3c
 00051908 ap_invoke_handler (1028e8, 84000, 1e7a48, 0, 0, fffffffe) + 78
 00034230 ap_process_request (1e7a48, a, 4, 1e7a48, 9a2e8, a) + 84
 0002f6b4 ap_process_http_connection (1d2c60, 1d2b70, 1d2b70, 4a, 9a2e8, 156458) + f4
 0005cb3c ap_run_process_connection (1d2c60, 1d2b70, 1d2b70, 4a, 1d2c58, 1d9a98) + 3c
 0004dfe8 process_socket (1d2b38, 1d2b70, 1, a, 1d9a98, e75f0) + 68
 0004e6c0 worker_thread (e7798, 99400, 0, 0, 0, 0) + 15c
 ff30487c dummy_worker (e7798, 0, 0, 0, 0, 0) + c
 fef85854 _lwp_start (0, 0, 0, 0, 0, 0)

-------------------------------------------------------------------------------------------------------------

Also, here is a snapshot of  the following command:
prstat -m -L -s cpu -p 427

   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
   427 www1      73 0.0 0.1 0.0 0.0 0.0 0.0  27   0 255   0   0 httpd/13
   427 www1      59 0.0 0.0 0.0 0.0 0.0 0.0  41   0 166   0   0 httpd/25
   427 www1     0.1 0.1 0.0 0.0 0.0  40  60 0.0  31   2  74   0 httpd/3
   427 www1     0.0 0.0 0.0 0.0 0.0  25  75 0.0   4   0   5   0 httpd/17
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   2   0 httpd/28
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/27
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/26
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/24
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/23
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 httpd/22
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/21
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/20
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/19
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/18
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/16
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/15
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/14
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/12
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/11
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/10
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/9
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/8
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/7
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/6
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/5
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 httpd/4
   427 www1     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 httpd/1
----------------------------------------------------------------------------------------------------------------------

Thank you

Mvh

Kjell Grindalen

This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal.


[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux