Hi all, We're having a problem with our Apache processes getting stuck inside an apr_poll() call. This is happening on a production box and it only seems to make itself a problem when the load gets up to > ~10 requests per second. We're using Apache 2.0.46 (RedHat patch, prefork version) in conjunction with PHP 4.3.2 (RedHat patchlevel 23) on a dual-processor 3.6 GHz machine. We have 2 gig of memory and a 36Mb RAID array, running under RedHat Linux AS3, kernel 2.4.21-32.ELsmp. In general, processes seem to get hung at random inside the apr_poll() call. I'm no Apache expert, but on a quick read of the stack trace, it looks like the system is trying to wait for the socket write to complete... except that it never does. We're using PHP sessions, so when one server process hangs, it leaves an exclusive lock open on the session file. If the user presses reload or attempts to access another page, the following HTTPD processes will block on accessing the session file. This quickly snowballs and puts us up past our MaxClients setting, which shuts down the server. I can't figure out how to reproduce the problem. It's taken out our server twice so far under heavy system load, but I'm unable to make it happen at will. Once, I was able to repro the bug from my own browser. On the client's side, it submitted a page request for "/", and it just hung there waiting for a response (and got nothing). If I wait during peak hours when lots of people are using the machine, I can usually see a couple cases of this from the server, but I've been strictly reactive so far. The PHP script and/or page being accessed does not seem to matter. We recently changed hardware and upgraded from Apache 1.3 to 2.0, so those could be potential factors. (However, we have two identical web servers that both exhibit the same problem, so a hardware failure is unlikely.) We might have seen a similar problem (with lots of HTTPD processes hanging around) under Apache 1.3 in the old environment. Unfortunately, our upgrade to new hardware also coincides with a heavy increase in traffic to our sites, so I can't necessarily point my finger one way or another. Can anyone make heads or trails out of the following GDB stack trace? I rebuilt libapr and httpd with symbols so I could see what was going on, and I managed to attach a debugger to the session and capture the following. Even if I let it run for a while, it stays stuck inside the poll() call: #0 0x00262689 in poll () from /lib/tls/libc.so.6 #1 0x00983c54 in apr_poll () from /usr/lib/libapr-0.so.0 #2 0x009843a2 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-0.so.0 #3 0x00979d73 in apr_socket_sendv () from /usr/lib/libapr-0.so.0 #4 0x0097a244 in apr_sendv () from /usr/lib/libapr-0.so.0 #5 0x0807bb65 in writev_it_all (s=0x9a80e20, vec=0xbfffa990, nvec=2, len=44839, nbytes=0xbfffa8e8) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/core.c:2864 #6 0x0807d038 in core_output_filter (f=0x9a812b0, b=0x9cca288) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/core.c:4170 #7 0x0807483c in ap_pass_brigade (next=0x7531, bb=0x1) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/util_filter.c:541 #8 0x08062d11 in ap_http_header_filter (f=0xa046638, b=0x9a8bfb0) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/modules/http/http_protocol.c:1716 #9 0x0807483c in ap_pass_brigade (next=0x7531, bb=0x1) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/util_filter.c:541 #10 0x08076edf in ap_content_length_filter (f=0xa046620, b=0x9a8bfb0) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/protocol.c:1270 #11 0x0807483c in ap_pass_brigade (next=0x7531, bb=0x1) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/util_filter.c:541 #12 0x08064aa4 in ap_byterange_filter (f=0xa046608, bb=0x9a8bfb0) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/modules/http/http_protocol.c:3001 #13 0x0807483c in ap_pass_brigade (next=0x7531, bb=0x1) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/util_filter.c:541 #14 0x0114fa2b in execute () from /etc/httpd/modules/libphp4.so #15 0x01122195 in php_ob_get_length () from /etc/httpd/modules/libphp4.so #16 0x0112222f in php_ob_get_length () from /etc/httpd/modules/libphp4.so #17 0x0112143f in php_end_ob_buffer () from /etc/httpd/modules/libphp4.so #18 0x01121790 in php_end_ob_buffers () from /etc/httpd/modules/libphp4.so #19 0x011108e8 in php_request_shutdown () from /etc/httpd/modules/libphp4.so #20 0x01150443 in execute () from /etc/httpd/modules/libphp4.so #21 0x011506a8 in execute () from /etc/httpd/modules/libphp4.so #22 0x0807483c in ap_pass_brigade (next=0x7531, bb=0x1) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/util_filter.c:541 #23 0x0807c489 in default_handler (r=0x11f1818) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/core.c:3555 #24 0x080685c5 in ap_run_handler (r=0xa045960) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/config.c:200 #25 0x08068bdf in ap_invoke_handler (r=0xa045960) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/config.c:410 #26 0x08065266 in rename_original_env (p=0xa045960, t=0xfffffffc) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/modules/http/http_request.c:317 #27 0x080608bc in ap_process_http_connection (c=0x99450c8) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/modules/http/http_core.c:293 #28 0x08072195 in ap_run_process_connection (c=0x9a80ef8) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/connection.c:85 #29 0x08066ae1 in child_main (child_num_arg=-4) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/mpm/prefork/prefork.c:706 #30 0x08066c34 in make_child (s=0x7531, slot=6) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/mpm/prefork/prefork.c:749 #31 0x08066d56 in perform_idle_server_maintenance (p=0x8) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/mpm/prefork/prefork.c:841 #32 0x0806757d in ap_mpm_run (_pconf=0x97bf0a8, plog=0x97e9150, s=0x97bf0a8) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/mpm/prefork/prefork.c:983 #33 0x0806da4f in main (argc=1, argv=0xbfffb614) at /usr/local/src/sdudley/RPM/BUILD/httpd-2.0.46/server/main.c:671 == Also, in case it matters, here is what the scoreboard looks like when the system gets wedged. We have plenty of memory and CPU cycles left when this happens. Apache Server Status for example.com Server Version: Apache Server Built: Feb 25 2005 09:35:45 Current Time: Thursday, 08-Sep-2005 15:01:42 CDT Restart Time: Thursday, 08-Sep-2005 14:40:49 CDT Parent Server Generation: 0 Server uptime: 20 minutes 53 seconds Total accesses: 15303 - Total Traffic: 163.1 MB CPU Usage: u869.65 s79.58 cu.29 cs.44 - 75.8% CPU load 12.2 requests/sec - 133.3 kB/second - 10.9 kB/request 150 requests currently being processed, 0 idle workers WWWWWWWWWWWWWWWWWWWWWWCWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWW.......................................... ................................................................ Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 0-0 14767 0/140/554 W 9.24 60 0 0.0 1.40 5.57 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 1-0 14861 0/4/126 W 0.83 19 0 0.0 0.08 2.33 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 2-0 14092 0/49/49 W 18.10 1036 0 0.0 0.35 0.35 65.162.244.194 example.com GET /projects/viewProject.php?_mode=search&_clearFlag=course,sp 3-0 14862 0/4/556 W 1.03 20 0 0.0 0.10 4.59 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 4-0 14094 0/605/605 W 71.52 55 0 0.0 7.80 7.80 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 5-0 14386 0/541/548 W 36.02 46 0 0.0 5.89 5.90 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 6-0 14863 0/2/53 W 0.63 30 0 0.0 0.05 0.75 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 7-0 14097 0/260/260 W 26.50 48 0 0.0 2.20 2.20 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 8-0 14098 0/216/216 W 37.65 6 0 0.0 4.00 4.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 9-0 14589 0/356/524 W 39.67 55 0 0.0 3.79 5.60 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 10-0 14864 0/5/169 W 0.73 45 0 0.0 0.05 1.83 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 11-0 14218 0/632/638 W 58.11 0 0 0.0 7.82 7.88 66.190.215.152 example.com GET / HTTP/1.1 12-0 14102 0/47/47 W 3.44 49 0 0.0 0.60 0.60 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 13-0 14865 0/3/138 W 0.44 61 0 0.0 0.18 1.64 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 14-0 14866 0/5/249 W 0.78 3 0 0.0 0.08 4.70 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 15-0 14105 0/608/608 W 60.14 13 0 0.0 6.56 6.56 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 16-0 14280 0/583/587 W 56.92 44 0 0.0 5.10 5.12 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 17-0 14867 0/3/22 W 0.65 31 0 0.0 0.06 0.46 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 18-0 14868 0/5/105 W 1.72 0 0 0.0 0.12 0.98 209.223.94.229 example.com GET /voodoo-status HTTP/1.0 19-0 14283 0/565/575 W 51.57 52 0 0.0 5.97 6.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 20-0 14869 0/2/173 W 0.54 8 0 0.0 0.05 1.60 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 21-0 14111 0/668/668 W 49.32 36 0 0.0 5.97 5.97 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 22-0 14870 1/11/20 C 1.89 0 0 11.4 0.16 0.33 66.83.74.234 example.com GET /uploaded_files/deliverFile.php/hon%20icon.jpg HTTP/1.1 23-0 14871 0/6/8 W 1.02 43 0 0.0 0.06 0.07 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 24-0 14872 0/3/4 W 0.35 53 0 0.0 0.03 0.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 25-0 14873 0/1/371 W 0.29 56 0 0.0 0.03 4.66 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 26-0 14874 0/2/14 W 0.58 5 0 0.0 0.05 0.10 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 27-0 14875 0/4/15 W 1.06 0 0 0.0 0.07 0.17 66.18.105.194 example.com GET /r.php/1120902432 HTTP/1.1 28-0 14876 0/5/6 W 0.53 53 0 0.0 0.25 0.25 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 29-0 14877 0/1/4 W 0.31 46 0 0.0 0.03 0.13 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 30-0 14129 0/632/632 W 58.34 59 0 0.0 6.87 6.87 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 31-0 14878 0/5/104 W 0.63 43 0 0.0 0.05 1.35 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 32-0 14879 0/3/10 W 0.42 47 0 0.0 0.04 0.06 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 33-0 14880 0/4/12 W 0.92 39 0 0.0 0.08 0.14 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 34-0 14881 0/2/19 W 0.65 42 0 0.0 0.05 0.13 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 35-0 14138 0/81/81 W 5.14 56 0 0.0 1.04 1.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 36-0 14882 0/2/14 W 0.42 49 0 0.0 0.04 0.30 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 37-0 14883 0/1/18 W 0.30 50 0 0.0 0.03 0.07 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 38-0 14145 0/669/669 W 50.81 47 0 0.0 6.89 6.89 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 39-0 14884 0/2/8 W 0.40 45 0 0.0 0.18 0.22 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 40-0 14885 0/5/111 W 0.77 37 0 0.0 0.05 1.21 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 41-0 14886 0/6/30 W 0.54 50 0 0.0 0.06 0.27 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 42-0 14149 0/1/1 W 0.00 1235 0 0.0 0.0000 0.0000 64.132.138.66 example.com POST /login/login.php HTTP/1.1 43-0 14150 0/634/634 W 56.34 57 0 0.0 5.24 5.24 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 44-0 14887 0/3/29 W 0.61 41 0 0.0 0.05 0.23 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 45-0 14888 0/2/6 W 0.51 39 0 0.0 0.04 0.10 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 46-0 14154 0/0/0 W 0.00 1232 0 0.0 0.00 0.00 64.132.138.66 example.com POST /login/login.php HTTP/1.1 47-0 14155 0/253/253 W 14.58 38 0 0.0 2.02 2.02 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 48-0 14889 0/2/117 W 0.43 51 0 0.0 0.04 0.95 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 49-0 14890 0/2/11 W 0.34 52 0 0.0 0.03 0.06 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 50-0 14891 0/2/22 W 0.34 47 0 0.0 0.03 0.12 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 51-0 14894 0/1/59 W 0.32 35 0 0.0 0.03 0.81 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 52-0 14895 0/1/328 W 0.30 33 0 0.0 0.03 2.40 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 53-0 14896 0/2/46 W 0.32 32 0 0.0 0.03 0.54 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 54-0 14897 0/1/20 W 0.29 28 0 0.0 0.03 0.22 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 55-0 14898 0/1/10 W 0.30 26 0 0.0 0.03 0.10 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 56-0 14899 0/1/196 W 0.33 27 0 0.0 0.03 2.12 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 57-0 14900 0/1/328 W 0.27 30 0 0.0 0.03 2.98 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 58-0 14901 0/1/89 W 0.28 21 0 0.0 0.03 0.85 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 59-0 14902 0/1/22 W 0.35 18 0 0.0 0.03 0.10 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 60-0 14168 0/598/598 W 54.77 54 0 0.0 7.50 7.50 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 61-0 14903 0/1/124 W 0.33 24 0 0.0 0.03 1.94 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 62-0 14173 0/713/713 W 59.91 38 0 0.0 7.55 7.55 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 63-0 14904 0/2/11 W 0.43 17 0 0.0 0.05 0.07 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 64-0 14905 0/1/6 W 0.33 22 0 0.0 0.03 0.05 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 65-0 14906 0/1/104 W 0.28 24 0 0.0 0.03 0.85 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 66-0 14178 0/425/425 W 41.45 51 0 0.0 4.58 4.58 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 67-0 14907 0/0/5 W 0.42 71 0 0.0 0.00 0.05 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 68-0 14908 0/1/2 W 0.32 25 0 0.0 0.03 0.05 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 69-0 14909 0/2/87 W 0.37 10 0 0.0 0.03 0.95 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 70-0 14910 0/1/7 W 0.35 16 0 0.0 0.03 0.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 71-0 14911 0/2/64 W 0.61 5 0 0.0 0.04 0.91 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 72-0 14912 0/1/154 W 0.28 14 0 0.0 0.03 1.10 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 73-0 14913 0/1/483 W 0.33 12 0 0.0 0.03 4.37 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 74-0 14186 0/592/592 W 44.76 9 0 0.0 5.73 5.73 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 75-0 14914 0/1/1 W 0.31 11 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 76-0 14915 0/0/0 W 0.00 70 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 77-0 14916 0/1/1 W 0.35 10 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 78-0 14917 0/1/1 W 0.28 9 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 79-0 14918 0/0/0 W 0.00 70 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 80-0 14919 0/7/7 W 0.72 42 0 0.0 0.07 0.07 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 81-0 14920 0/1/1 W 0.29 7 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 82-0 14921 0/1/1 W 0.27 7 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 83-0 14922 0/1/1 W 0.33 8 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 84-0 14923 0/6/6 W 1.06 1 0 0.0 0.11 0.11 139.76.128.68 example.com GET /uploaded_files/deliverFile.php/playground%20hurricane.gif 85-0 14924 0/1/1 W 0.34 6 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 86-0 14925 0/2/2 W 0.33 13 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 87-0 14926 0/3/3 W 0.43 31 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 88-0 14927 0/2/2 W 0.31 14 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 89-0 14928 0/5/5 W 0.48 29 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 90-0 14929 0/1/1 W 0.32 29 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 91-0 14930 0/1/1 W 0.32 11 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 92-0 14931 0/1/1 W 0.27 28 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 93-0 14932 0/1/1 W 0.32 26 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 94-0 14933 0/1/1 W 0.31 27 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 95-0 14934 0/2/2 W 0.33 16 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 96-0 14935 0/6/6 W 0.49 34 0 0.0 0.07 0.07 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 97-0 14936 0/1/1 W 0.27 19 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 98-0 14937 0/6/6 W 0.56 43 0 0.0 0.04 0.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 99-0 14938 0/8/8 W 0.44 41 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 100-0 14939 0/4/4 W 0.10 67 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 101-0 14940 0/2/2 W 0.28 17 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 102-0 14941 0/1/1 W 0.31 25 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 103-0 14942 0/6/6 W 0.70 37 0 0.0 0.06 0.06 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 104-0 14943 0/5/5 W 0.44 34 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 105-0 14944 0/3/3 W 0.34 15 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 106-0 14945 0/4/4 W 0.53 32 0 0.0 0.05 0.05 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 107-0 14946 0/2/2 W 0.28 22 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 108-0 14947 0/1/1 W 0.29 15 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 109-0 14948 0/1/1 W 0.30 23 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 110-0 14949 0/1/1 W 0.28 21 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 111-0 14950 0/4/4 W 0.39 35 0 0.0 0.04 0.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 112-0 14951 0/7/7 W 0.71 36 0 0.0 0.05 0.05 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 113-0 14952 0/2/2 W 0.36 18 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 114-0 14953 0/1/1 W 0.35 20 0 0.0 0.03 0.03 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 115-0 14954 0/2/2 W 0.43 33 0 0.0 0.04 0.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 116-0 14955 0/5/5 W 0.44 39 0 0.0 0.05 0.05 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 117-0 14956 0/5/5 W 0.67 4 0 0.0 0.04 0.04 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 118-0 14957 0/5/5 W 0.09 65 0 0.0 0.01 0.01 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 119-0 14958 0/3/3 W 0.64 2 0 0.0 0.05 0.05 152.130.8.6 example.com GET /projects/viewProject.php?_mode=occurrenceView&_action=load 120-0 14959 0/1/1 W 0.42 0 0 0.0 0.03 0.03 65.87.105.1 example.com GET /projects/viewProject.php?_Offset=10&_mode=browse&_action=f 121-0 14960 0/0/0 W 0.00 64 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 122-0 14961 0/0/0 W 0.00 64 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 123-0 14962 0/0/0 W 0.00 64 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 124-0 14963 0/1/1 W 0.00 64 0 0.0 0.0000 0.0000 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 125-0 14964 0/0/0 W 0.00 63 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 126-0 14965 0/0/0 W 0.00 63 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 127-0 14966 0/4/4 W 0.34 62 0 0.0 0.20 0.20 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 128-0 14967 0/1/1 W 0.01 63 0 0.0 0.0000 0.0000 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 129-0 14968 0/0/0 W 0.00 63 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 130-0 14969 0/0/0 W 0.00 63 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 131-0 14970 0/1/1 W 0.00 63 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 132-0 14971 0/2/2 W 0.28 62 0 0.0 0.02 0.02 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 133-0 14972 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 134-0 14973 0/1/1 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 135-0 14974 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 136-0 14975 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 137-0 14976 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 138-0 14977 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 139-0 14978 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 140-0 14979 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 141-0 14980 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 142-0 14981 0/1/1 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 143-0 14982 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 144-0 14983 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 145-0 14984 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 146-0 14985 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 147-0 14986 0/1/1 W 0.08 61 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 148-0 14987 0/0/0 W 0.00 62 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ 149-0 14988 0/0/0 W 0.00 61 0 0.0 0.00 0.00 65.168.18.47 example.com GET /projects/viewProject.php?_Offset=10&_mode=occurrenceView&_ Srv Child Server number - generation PID OS process ID Acc Number of accesses this connection / this child / this slot M Mode of operation CPU CPU usage, number of seconds SS Seconds since beginning of most recent request Req Milliseconds required to process most recent request Conn Kilobytes transferred this connection Child Megabytes transferred this child Slot Total megabytes transferred this slot Apache Server at www.example.com Port 80 Thanks! SD ______________________________________________________ Click here to donate to the Hurricane Katrina relief effort. http://store.yahoo.com/redcross-donate3/ --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx