I need some expert assistance to pinpoint the cause of intermittent hangs affecting high-traffic web servers running Apache 2.0.52 & ColdFusion MX 6.1 under Windows. I've been working for months on this issue with senior engineers at Macromedia and more recently with engineers at Microsoft. This issue has stumped all of them -- no one has been able to pin down the cause, and both companies are now telling me that they think the issue is in Apache and that I should engage an Apache consulting resource. I'd love to do that, but I don't know where to find such a resource. Below is a summary of the issue. Can anyone on this list offer assistance, or point me to someone who might be able to help? Thanks, Laura ---------------------------------------- PLATFORM: Windows 2000 / 2003 (all with latest SP and hotfixes) Apache 2.0.52 ColdFusion MX 6.1 All web sites are running a proprietary ecommerce application written by our developers. All pages on every site are database-driven and are served through Apache/ColdFusion. We do not allow 3rd-party ColdFusion code on any of our servers, nor do we support any other server-side processing. SUMMARY: Intermittently, all web sites hosted on any given server will hang for somewhere between 10 seconds and 10+ minutes. If left alone long enough, the hang usually resolves on its own, but sometimes requires restart of Apache & ColdFusion services, and occasionally a server reboot. Behavior is unpredictable and cannot be reproduced in a development environment, but happens on all production servers (happens more frequently under heavy load). Each server may go a day or more with no problem, or may hang repeatedly several days in row. During the hung state: (1) Apache can serve plain text files, but not ColdFusion pages (2) ColdFusion can serve dynamic database-driven pages on its internal web server on port 8500. The Macromedia engineer I've been working with tells me that when a hang occurs, an Apache thread has made a blocking call to WSASend, and this socket gets "stuck" for some period of time (anywhere from 10 seconds to 10+ minutes). They had me open a ticket with Microsoft to try to determine exactly what WSASend is waiting on in the thread below (collected during a hang). Microsoft says they are "unable to provide this information", and that as far as they are concerned, the problem is in either Apache or ColdFusion. Macromedia can't find any evidence of a problem in ColdFusion causing these hangs. # ChildEBP RetAddr Args to Child 00 010af9b0 74fd1394 000006d8 00000001 010af9d8 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) 01 010af9ec 74fdaa24 000006d8 00001d28 00000000 msafd!SockWaitForSingleObject+0x1a8 (FPO: [Non-Fpo]) 02 010afa68 75031591 00001d28 010afac8 00000001 msafd!WSPSend+0x1be (FPO: [Non-Fpo]) 03 010afaa4 6eecb08f 00001d28 010afac8 00000001 WS2_32!WSASend+0x8a (FPO: [Non-Fpo]) 04 010afae4 6ff0781b 0091b660 010afb84 00000001 libapr!apr_socket_sendv(struct apr_socket_t * sock = 0x0091b660, struct iovec * vec = 0x010afb84, int nvec = 1, unsigned int * nbytes = 0x010afb08)+0x8f (FPO: [Non-Fpo]) (CONV: stdcall) [d:\httpd-2.0.52\srclib\apr\network_io\win32\sendrecv.c @ 115] 05 010afb08 6ff08c44 0091b660 00000001 00002000 libhttpd!writev_it_all(struct apr_socket_t * s = <Memory access error>, struct iovec * vec = 0x010afaec, int nvec = <Memory access error>, unsigned int len = <Memory access error>, unsigned int * nbytes = <Memory access error>)+0x3b (FPO: [EBP 0x008ccbe0] [4,1,0]) (CONV: cdecl) [d:\httpd-2.0.52\server\core.c @ 2920] 06 010afc80 6ff186d8 0091ba88 03c0e5d8 03c0e5d8 libhttpd!core_output_filter(struct ap_filter_t * f = 0x03c0e5dc, struct apr_bucket_brigade * b = 0x6ee7f300)+0x594 (FPO: [EBP 0x03c20020] [2,85,0]) (CONV: cdecl) [d:\httpd-2.0.52\server\core.c @ 4237] 07 010afc90 6ff0fb0e 0091ba88 03c0e5d8 6ff187a0 libhttpd!ap_pass_brigade(struct ap_filter_t * next = 0x0091b730, struct apr_bucket_brigade * bb = 0x008b0668)+0x58 (FPO: [2,0,0]) (CONV: stdcall) [d:\httpd-2.0.52\server\util_filter.c @ 511] 08 010afcbc 6ff186d8 03c20cb8 03c0e5dc 03c0e5d8 libhttpd!ap_content_length_filter(struct ap_filter_t * f = 0x0091b730, struct apr_bucket_brigade * b = 0x008b0668)+0x19e (FPO: [EBP 0x00002000] [2,4,0]) (CONV: cdecl) [d:\httpd-2.0.52\server\protocol.c @ 1256] 09 010afccc 6ff187af 03c20cb8 03c0e5d8 6ee614d4 libhttpd!ap_pass_brigade(struct ap_filter_t * next = <Memory access error>, struct apr_bucket_brigade * bb = <Memory access error>)+0x58 (FPO: [2,0,0]) (CONV: stdcall) [d:\httpd-2.0.52\server\util_filter.c @ 511] 0a 010afcd8 6ee614d4 03c0e5d8 03c20cb8 03c0e4a8 libhttpd!ap_filter_flush(struct apr_bucket_brigade * bb = <Memory access error>, void * ctx = <Memory access error>)+0xf (FPO: [2,0,0]) (CONV: cdecl) [d:\httpd-2.0.52\server\util_filter.c @ 549] 0b 010afcf4 6ff0fd75 03c0e5d8 6ff187a0 03c20cb8 libaprutil!apr_brigade_write(struct apr_bucket_brigade * b = <Memory access error>, <function> * flush = <Memory access error>, void * ctx = <Memory access error>, char * str = <Memory access error>, unsigned int nbyte = <Memory access error>)+0x74 (FPO: [EBP 0x0091b730] [5,0,0]) (CONV: stdcall) [d:\httpd-2.0.52\srclib\apr-util\buckets\apr_brigade.c @ 392] 0c 010afd18 6ff0fe35 063d3068 00002000 00002000 libhttpd!buffer_output(struct request_rec * r = 0x00000102, char * str = 0x008b0668 "X}???", unsigned int len = 0)+0xd5 (FPO: [EBP 0xffffffff] [2,0,0]) (CONV: cdecl) [d:\httpd-2.0.52\server\protocol.c @ 1373] 0d 010afd2c 00f77fb3 063d3068 00002000 03c20020 libhttpd!ap_rwrite(void * buf = <Memory access error>, int nbyte = <Memory access error>, struct request_rec * r = <Memory access error>)+0x25 (FPO: [3,0,0]) (CONV: stdcall) [d:\httpd-2.0.52\server\protocol.c @ 1408] WARNING: Stack unwind information not available. Following frames may be wrong. 0e 010afd48 00f742f7 010afde4 063d3068 00002000 mod_jrun20+0x7fb3 0f 063d3068 0909090a 09090909 090a0d09 0a0d0909 mod_jrun20+0x42f7 10 0d090909 00000000 00000000 00000000 00000000 0x909090a --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|