Hello,
My apologizes, my post of yesterday included HTML tables from server-status output.(cut/paste).
Here is a more digest version.
Our httpd Apache server keeps hanging every day or many times a day depending on the activity.
Apache is the frontend of our Subversion 1.6.5 server.
We have about a 100 client developers accessing SVN thru Apache and a few Hudson continuous integration servers accessing SVN as well and running many automatic SVN accesses 24/7.
SVN is working well because when the problem happens, I can isssue local SVN commands outside of httpd without any problem.
Our Apache is listening to 2 different ports 20000 and 20001.
Port 20000 is the standard SVN port and port 20001 is used by the SVN PHP Admin tools. (svnadmin and websvn)
When the httpd Apache hangs it is always on port 20000.
All SVN requests are stuck for some reasons.
When the httpd hangs on port 20000, if we issue an http request using svnadmin on port 20001 then the server is unblocked for a short moment sometime few minutes after hanging again.
Then if we re-issue a request on port 20001 it will again release port 20000 requests for a while.
The only clean solution is to restart the httpd server.
Then it will eventualy run for a few hours or a day or two and then hang again.
We turned keepalive to OFF to test if we were hiting socket release issues but the problem just happened with KEEPALIVE set to OFF.
Our server is a large solaris machine with plenty of memory and swap.
Solaris : SunOS MyMachine 5.10 Generic_147440-24 sun4v sparc sun4v
Apache: Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2 SVN/1.6.5 PHP/5.3.1
------------------------------------------------------------------------
Here is the extract from our httpd.conf file
ServerRoot "/opt/xxxxxx/xxx/xxxxxxxxxxxxxxxxxxxxxxx"
LockFile accept.lock
PidFile apache2.pid
Listen
0.0.0.0:20001LockFile accept.lock
PidFile apache2.pid
Timeout 300
KeepAlive Off # was On before
MaxKeepAliveRequests 1536
KeepAliveTimeout 2
<IfModule mpm_prefork_module>
StartServers 32
MinSpareServers 48
MaxSpareServers 64
ServerLimit 512
MaxClients 96
MaxRequestsPerChild 1536
</IfModule>
[...]
<IfModule alias_module>
ScriptAlias /cgi-bin/ "/opt/blablabla/cgi-bin/"
</IfModule>
<Directory "/opt/blablabla/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
# Main server Admin SVN
ServerName
scm.xyz.com
DocumentRoot "/opt/blablabla/htdocs"
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory "/opt/blablabla/htdocs">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
# PHP5 :
Include conf/extra/httpd-php.conf
# Repository SVN
Listen
0.0.0.0:20000
<VirtualHost *:20000>
Include conf/extra/httpd-svn.conf
</VirtualHost>
------------------------------------------------------------------------
After the apache is hung here is the status-server request:
Apache Server Status for
scm.xyz.comServer Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2 SVN/1.6.5 PHP/5.3.1
Server Built: Jan 13 2010 15:49:24
Current Time: Friday, 16-Aug-2013 15:21:52 CEST
Restart Time: Friday, 16-Aug-2013 03:10:02 CEST
Parent Server Generation: 0
Server uptime: 12 hours 11 minutes 50 seconds
Total accesses: 228561 - Total Traffic: 369.5 MB
CPU Usage: u637.01 s183.88 cu1.4 cs0 - 1.87% CPU load
5.21 requests/sec - 8.6 kB/second - 1694 B/request
8 requests currently being processed, 55 idle workers
_______________________________WWW_W___W______WWR______________.
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 20113 0/751/3794 _ 7.48 606 6 0.0 0.49 2.95 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
1-0 27522 0/640/3674 _ 12.28 627 7 0.0 0.49 3.04 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1
2-0 20194 0/735/3776 _ 7.80 627 6 0.0 0.42 4.36 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
3-0 27523 0/647/3685 _ 44.08 442 162540 0.0 20.14 21.82 xx.yyy.yy.108 myserver.a.b.c REPORT /my.path/!svn/blablabla/default HTTP/1.1
4-0 3076 0/1391/3425 W 14.89 0 0 0.0 0.90 2.42 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.
5-0 10815 0/397/3735 _ 4.42 627 18 0.0 0.24 3.12 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/bblablabla/trunk H
6-0 27524 0/643/3682 _ 13.14 606 33 0.0 1.68 3.58 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/vcc/default HTTP/1.1
7-0 28908 0/633/3666 _ 7.06 627 9 0.0 0.42 5.62 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/vcc/default HTTP/1.1
8-0 27525 0/640/3673 _ 13.35 627 9 0.0 4.97 7.44 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
9-0 27527 0/639/3675 _ 27.84 627 11 0.0 2.62 5.38 xx.yyy.yy.114 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
10-0 27528 0/638/3677 _ 8.09 606 3 0.0 0.60 7.14 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla HTTP/1.1
11-0 16355 0/301/3469 _ 3.08 627 6 0.0 0.18 4.83 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
12-0 2761 0/989/3282 _ 10.41 627 6 0.0 0.67 2.72 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
13-0 29771 0/632/3667 _ 6.51 627 16 0.0 0.38 3.40 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HT
14-0 29928 0/586/3621 _ 8.50 627 5 0.0 0.49 10.14 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
15-0 29772 0/629/3666 _ 6.77 606 8 0.0 0.37 3.15 xx.yyy.yy.108 myserver.a.b.c REPORT /my.path/!svn/blablabla HTTP/1.1
16-0 27529 0/642/3682 _ 6.56 627 6 0.0 0.36 3.67 xx.yyy.yy.114 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
17-0 27530 0/631/3672 _ 11.19 606 6 0.0 0.43 3.80 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
18-0 27827 0/641/3678 _ 6.79 615 5 0.0 0.38 2.20 xx.yyy.yy.140 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
19-0 28230 0/640/3098 _ 7.22 627 6 0.0 0.39 1.72 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
20-0 28302 0/639/3679 _ 7.11 606 6 0.0 0.48 4.59 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
21-0 28231 0/639/3675 W 8.22 0 0 0.0 0.60 4.22 xx.yyy.yy.5 myserver.a.b.c GET /my.path/ HTTP/1.1
22-0 3077 0/1373/2899 _ 26.72 627 17 0.0 3.16 10.28 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk
23-0 20679 0/235/3408 _ 2.49 606 15 0.0 0.13 2.20 xx.yyy.yy.108 myserver.a.b.c REPORT /my.path/!svn/blablabla/default HTTP/1.1
24-0 3078 0/1386/2905 _ 16.83 627 14 0.0 2.16 3.08 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablablat/trunk
25-0 28297 0/616/2131 _ 7.94 627 11 0.0 0.59 11.05 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
26-0 28299 0/634/3674 _ 8.10 627 15 0.0 0.53 4.19 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk
27-0 28300 0/641/2157 _ 6.58 606 5 0.0 0.34 1.12 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
28-0 28303 0/642/2159 _ 6.82 615 5 0.0 0.43 1.61 xx.yyy.yy.140 myserver.a.b.c PROPFIND /my.path HTTP/1.1
29-0 28304 0/639/3673 _ 7.29 615 10 0.0 0.50 7.08 xx.yyy.yy.140 myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
30-0 28305 0/633/2149 W 7.62 0 0 0.0 0.44 3.47 xx.yyy.yy.119 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
31-0 29929 0/585/3619 W 6.31 0 0 0.0 0.41 4.00 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
32-0 29930 0/592/3627 W 7.45 0 0 0.0 0.55 2.51 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.
33-0 28910 0/635/3673 W 8.87 0 0 0.0 0.38 2.44 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
34-0 28696 0/641/3676 _ 6.57 606 98 0.0 0.38 10.64 xx.yyy.yy.108 myserver.a.b.c REPORT /my.path/!svn/blablabla/default HTTP/1.1
35-0 6557 0/481/3517 W 4.98 0 0 0.0 0.33 8.89 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk H
36-0 28700 0/633/3671 _ 7.61 627 6 0.0 0.38 2.22 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
37-0 29931 0/588/3621 _ 6.22 627 14 0.0 0.36 13.57 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/testsSecurity
38-0 28697 0/626/3661 _ 9.76 606 71 0.0 0.86 3.63 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
39-0 29766 0/628/3662 W 6.74 0 0 0.0 0.42 4.28 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/trunk/ HTTP/1.1
40-0 29932 0/586/3622 _ 7.25 606 4 0.0 0.40 2.32 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
41-0 13813 0/366/3616 _ 3.58 606 6 0.0 0.22 5.97 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
42-0 9 0/570/3606 _ 5.83 627 13 0.0 0.37 21.85 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla HTTP/1.1
43-0 29770 0/634/3668 _ 7.88 627 11 0.0 0.51 4.62 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
44-0 29916 0/614/3651 _ 8.82 627 16 0.0 2.77 4.50 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
45-0 29905 0/631/3670 _ 6.87 627 8 0.0 0.37 6.40 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
46-0 29906 0/623/3658 W 7.33 0 0 0.0 0.42 5.68 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1
47-0 29907 0/627/2494 W 7.11 0 0 0.0 0.51 1.57 xx.yyy.yy.243 myserver.a.b.c GET /server-status HTTP/1.1
48-0 29917 0/615/3648 W 6.73 0 0 0.0 1.35 4.79 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1
49-0 29918 0/607/3640 _ 9.02 606 6 0.0 0.79 3.13 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
50-0 29919 0/614/3648 _ 6.30 627 17 0.0 0.35 2.08 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/
51-0 29933 0/587/3620 W 8.42 0 0 0.0 0.41 2.94 xx.yyy.yy.5 myserver.a.b.c GET /my.path/trunk/ HTTP/1.1
52-0 29934 0/583/3618 _ 7.54 606 6 0.0 0.47 3.41 xx.yyy.yy.108 myserver.a.b.c PROPFIND /blablabla/trunk HTTP/1.1
53-0 29935 0/593/3630 _ 7.32 615 5 0.0 0.64 3.50 xx.yyy.yy.140 myserver.a.b.c PROPFIND blablabla HTTP/1.1
54-0 6566 0/467/3504 W 5.79 0 0 0.0 0.28 2.77 xx.yyy.yy.140 myserver.a.b.c PROPFIND /blablabla HTTP/1.1
55-0 6567 0/473/3508 _ 4.58 606 6 0.0 0.27 2.09 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
56-0 27600 0/132/1651 _ 1.71 627 9 0.0 0.14 7.04 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
57-0 27616 0/63/1602 _ 29.34 615 5 0.0 19.80 22.77 xx.yyy.yy.140 myserver.a.b.c PROPFIND /my.path HTTP/1.1
58-0 27617 0/118/1288 _ 1.65 627 6 0.0 0.08 0.86 xx.yyy.yy.114 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
59-0 27618 0/101/1277 W 0.96 0 0 0.0 0.07 3.19 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
60-0 27619 0/101/3137 _ 1.22 627 5 0.0 0.08 2.15 xx.yyy.yy.114 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
61-0 27621 0/101/3133 _ 1.01 615 5 0.0 0.06 1.77 xx.yyy.yy.140 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
62-0 27623 0/99/3133 _ 1.44 627 18 0.0 0.09 1.81 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla
63-0 - 0/0/1513 . 19.93 10227 11 0.0 0.00 1.34 xx.yyy.yy.108 myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
64-0 - 0/0/1513 . 16.68 10235 6 0.0 0.00 1.06 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
65-0 - 0/0/1511 . 51.59 10167 9 0.0 0.00 20.72 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/4389 HTTP/1.1
66-0 - 0/0/1520 . 17.51 4947 7 0.0 0.00 1.19 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1
67-0 - 0/0/1520 . 17.32 4998 7 0.0 0.00 1.32 xx.yyy.yy.30 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
68-0 - 0/0/1520 . 29.50 4995 10 0.0 0.00 2.92 xx.yyy.yy.30 myserver.a.b.c REPORT /my.path/!svn/blablabla/branches HTTP/1.1
69-0 - 0/0/784 . 8.89 12044 0 0.0 0.00 0.50 127.0.0.1 myserver.a.b.c OPTIONS * HTTP/1.0
70-0 - 0/0/1524 . 17.60 4849 46 0.0 0.00 2.01 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
71-0 - 0/0/1520 . 28.70 4850 18 0.0 0.00 2.42 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/
72-0 - 0/0/1520 . 15.69 5015 7 0.0 0.00 0.89 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
73-0 - 0/0/1520 . 31.96 4955 19 0.0 0.00 8.35 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla
74-0 - 0/0/1521 . 18.82 4998 4 0.0 0.00 0.95 xx.yyy.yy.30 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
75-0 - 0/0/1520 . 16.71 5015 8 0.0 0.00 0.96 xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
------------------------------------------------------------------------
Any help appreciated.
/Nourrédine
