Hello, folks I am currently experiencing a really awkward problem with our production systems for which I have filed http://issues.eu.apache.org/bugzilla/show_bug.cgi?id=37118. The connection to the server is normal whereupon the server reads the request and starts processing it. Since we do not set the FollowSymlinks option, we see calls to lstat() for each of the path elements, and then nothing for a few seconds during which the thread has apparently been put to sleep. When it resumes processing (identified by the return from lwp_unpark()), it logs data to the debug log before opening the file and sending it using sendfile(). The system is not particularly busy as shown by the vmstat output below. We observe peaks of 20 connections (accepts) per second. The problem is not network related since it occurs also locally on the server using ApacheBench (ab). What surprises me is that the thread is put to sleep for, sometimes, long periods of time. The highest observed is 14 seconds. There is no evidence of the process being blocked by IO. The requested file (in our test scenario) is only a few hundred bytes and on local disk. The server is a reverse proxy serving content from J2EE backend servers. Reducing the number of threads per process from 100 to 20 yields no effect apart from higher memory footprint. procs memory page disk faults cpu r b w swap free si so pi po fr de sr m0 m1 m2 m3 in sy cs us sy id 0 0 0 14192736 6676888 0 0 18 2 2 0 0 0 0 0 0 434 2052 749 3 3 94 0 0 0 13926984 6297968 0 0 0 0 0 0 0 0 0 0 0 560 3771 1243 6 4 90 4 0 0 13926984 6297832 0 0 0 8 8 0 0 0 0 0 0 1126 8035 2157 13 9 78 0 0 0 13926984 6297728 0 0 0 0 0 0 0 0 0 0 0 873 6631 1966 10 5 86 0 0 0 13926984 6297592 0 0 0 0 0 0 0 0 0 0 0 1110 5554 1760 10 12 78 0 0 0 13926984 6297520 0 0 0 0 0 0 0 0 0 0 0 807 3668 1397 5 10 85 0 0 0 13926984 6297480 0 0 0 0 0 0 0 0 0 0 0 617 3079 1220 5 2 93 1 0 0 13925352 6293552 0 0 0 4 4 0 0 0 0 0 0 707 6147 1332 8 6 86 0 0 0 13925664 6293792 0 0 0 0 0 0 0 0 0 0 0 1067 8028 2173 17 7 76 0 0 0 13926608 6296824 0 0 0 92 76 0 0 0 0 0 0 1540 10980 1977 12 12 76 0 0 0 13925944 6295888 0 0 0 4 4 0 0 0 0 0 0 1532 15132 1963 14 10 76 0 1 0 13925952 6295920 0 0 0 0 0 0 0 0 0 0 0 848 9263 1127 3 5 92 0 0 0 13926608 6296624 0 0 0 0 0 0 0 0 0 0 0 1404 2692 1247 6 5 89 0 0 0 13926608 6296528 0 0 0 0 0 0 0 0 0 0 0 1410 5101 1634 8 4 88 3 0 0 13926600 6296376 0 0 0 4 4 0 0 0 0 0 0 1500 12413 3115 18 11 71 0 0 0 13926568 6296184 0 0 0 0 0 0 0 0 0 0 0 1031 5763 1917 8 6 86 2 0 0 13924352 6295632 0 0 0 0 0 0 0 0 0 0 0 1164 9517 2321 23 6 70 0 0 0 13924048 6294960 0 0 0 4 4 0 0 0 0 0 0 2436 15093 3567 24 14 62 0 0 0 13924240 6294808 0 0 0 0 0 0 0 0 0 0 0 1559 13327 3065 25 12 62 I am very eager for any hints or tips regarding what to look for other than what I have already done... Cheers, -ascs --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|