On Wed, Mar 27, 2013 at 9:42 AM, Marcin Wanat <marcin.wanat@xxxxxxxxx> wrote: > Hi, > > i have weird problem with newly configured server. > I have running Apache 2.2.16(prefork) with php(module) 5.2.14 running > on CentOS5 for two years on my old server and last week i decided to > move to new server (CentOS6) and upgrade Apache to 2.4.4(prefork) and > php to 5.4.13. > On old server everything is working fine but when i route this traffic > (A DNS record of my domain) to new server then strange things are > happening. > > For the first 30 seconds after httpd start everything is normal, and > then after 30 seconds about 25% of active connections are in "R" > (Reading Request) state. These connections are freezed in R state for > 10-25secs and then dissapear. > They are not logged in error_log even in trace6 level. > > server-status for such connection looks like: > PID Acc M CPU SS Req Conn Child Slot Client VHost > 22182 0/90/90 R 0.73 3 5 0.0 0.03 0.03 81.96.86.224 > > netstat for this IP shows: > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52168 TIME_WAIT - > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52202 TIME_WAIT - > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52254 TIME_WAIT - > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52201 TIME_WAIT - > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52255 ESTABLISHED 22182/httpd > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52169 TIME_WAIT - > tcp 0 0 ::ffff:95.211.131.XXX:80 > ::ffff:81.96.86.224:52203 TIME_WAIT - > > Server load is about 0.5 and total CPU usage ~5%. Server is 16core, > 32gb ram, 4xSAS hdd so it is not overloaded. > Network is 100mbps and usage of about 4mbps (only html content). I > have tested connection to few IPs which freeze in R state with mtr, > but no single packet loss were found. > > Server is handling about 100requests/sec in peek hours and 50req/sec > during night. > > Regardless of req/sec problem always occurs after about 30secs after > httpd start. > apachectl restart does not change anything. Only manually killing all > httpd processes and starting them from scratch can give 30seconds free > of R state connections. > > I was also trying to recompile httpd with event mpm - effect is the > same. Even when i remove all htdocs content and every request is 404, > Reading Request connections appears after 30secs. Was trying httpd > 2.2.24 and the effect was the same. > > I was even trying different kernels (Centos6 2.6.32-358.2.1.el6 and > newest one 3.8.4) but with no luck. > > These Reading Request connections are comming from different networks > all over the world. > > I was also trying to do some stress tests with ab -n 50000 -c 20 from > different servers but each time these tests completed without any > freeze. > > Some details: > [root@srv logs]# httpd -V > Server version: Apache/2.4.4 (Unix) > Server built: Mar 24 2013 01:43:25 > Server's Module Magic Number: 20120211:11 > Server loaded: APR 1.4.6, APR-UTIL 1.5.1 > Compiled using: APR 1.4.6, APR-UTIL 1.5.1 > Architecture: 64-bit > Server MPM: prefork > threaded: no > forked: yes (variable process count) > Server compiled with.... > -D APR_HAS_SENDFILE > -D APR_HAS_MMAP > -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) > -D APR_USE_SYSVSEM_SERIALIZE > -D APR_USE_PTHREAD_SERIALIZE > -D APR_HAS_OTHER_CHILD > -D AP_HAVE_RELIABLE_PIPED_LOGS > -D DYNAMIC_MODULE_LIMIT=256 > -D HTTPD_ROOT="/usr/local/httpd" > -D SUEXEC_BIN="/usr/local/httpd/bin/suexec" > -D DEFAULT_PIDLOG="logs/httpd.pid" > -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" > -D DEFAULT_ERRORLOG="logs/error_log" > -D AP_TYPES_CONFIG_FILE="conf/mime.types" > -D SERVER_CONFIG_FILE="conf/httpd.conf" > [root@srv logs]# httpd -l > Compiled in modules: > core.c > mod_so.c > http_core.c > prefork.c > > Loaded modules: > LoadModule php5_module modules/libphp5.so > LoadModule unixd_module modules/mod_unixd.so > LoadModule authz_host_module modules/mod_authz_host.so > LoadModule authz_core_module modules/mod_authz_core.so > LoadModule access_compat_module modules/mod_access_compat.so > LoadModule rewrite_module modules/mod_rewrite.so > LoadModule mime_module modules/mod_mime.so > LoadModule dir_module modules/mod_dir.so > LoadModule authn_core_module modules/mod_authn_core.so > LoadModule authn_file_module modules/mod_authn_file.so > LoadModule status_module modules/mod_status.so > LoadModule authz_user_module modules/mod_authz_user.so > LoadModule auth_basic_module modules/mod_auth_basic.so > > Prefork config: > StartServers 256 > MinSpareServers 256 > MaxSpareServers 512 > MaxRequestWorkers 1024 > MaxConnectionsPerChild 0 What are the equivalent settings in your 2.2 setup? > > > > Currently i am out of ideas what else could i check to trace this problem. > > Regards, > Marcin Wanat > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx > -- Born in Roswell... married an alien... http://emptyhammock.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|