Re: Apache httpd 2.2.12 hangs

[Nourredine Rouibah <nourredine.rouibah@xxxxxxxxx>]

Hi,

After doing some search I suspect our problems to be related with this issue : https://issues.apache.org/bugzilla/show_bug.cgi?id=47645

I say that because the problem started to happen as soon as we changed our Solaris HW to a new one.

Before that, we experienced http hang outs but once every 3 months and a restart did solve the issue.

Since we moved to the new HW (same Solaris version) we httpd hangs many times per day.

As I described, the port 20000 hangs and a request to port 20001 did release the httpd.

Can someone confirm that it worth trying to rebuild with the above patch ?

thanks,

/Nourrédine

On Mon, Aug 19, 2013 at 10:19 AM, Nourredine Rouibah <nourredine.rouibah@xxxxxxxxx> wrote:

Hello,

Our httpd Apache server keeps hanging every day or many times a day depending on the activity.
Apache is the frontend of our Subversion 1.6.5 server.
We have about a 100 client developers accessing SVN thru Apache and a few Hudson continuous integration servers accessing SVN as well and running many automatic SVN accesses 24/7.
SVN is working well because when the problem happens, I can isssue local SVN commands outside of httpd without any problem.

Our Apache is listening to 2 different ports 20000 and 20001.
Port 20000 is the standard SVN port and port 20001 is used by the SVN PHP Admin tools. (svnadmin and websvn)

When the httpd Apache hangs it is always on port 20000.
All SVN requests are stuck for some reasons.

When the httpd hangs on port 20000, if we issue an http request using svnadmin on port 20001 then the server is unblocked for a short moment sometime few minutes after hanging again.
Then if we re-issue a request on port 20001 it will again release port 20000 requests for a while.
The only clean solution is to restart the httpd server.
Then it will eventualy run for a few hours or a day or two and then hang again.

We turned keepalive to OFF to test if we were hiting socket release issues but the problem just happened with KEEPALIVE set to OFF.
Our server is a large solaris machine with plenty of memory and swap.
Solaris : SunOS MyMachine 5.10 Generic_147440-24 sun4v sparc sun4v
Apache: Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2 SVN/1.6.5 PHP/5.3.1

After the apache is hung here is the status-server request:

Apache Server Status for scm.xyz.com
Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2 SVN/1.6.5 PHP/5.3.1
Server Built: Jan 13 2010 15:49:24
Current Time: Friday, 16-Aug-2013 15:21:52 CEST
Restart Time: Friday, 16-Aug-2013 03:10:02 CEST
Parent Server Generation: 0
Server uptime: 12 hours 11 minutes 50 seconds

Total accesses: 228561 - Total Traffic: 369.5 MB
CPU Usage: u637.01 s183.88 cu1.4 cs0 - 1.87% CPU load
5.21 requests/sec - 8.6 kB/second - 1694 B/request
8 requests currently being processed, 55 idle workers

_______________________________WWW_W___W______WWR______________.
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv	PID	Acc	M	CPU	SS	Req	Conn	Child	Slot	Client	VHost	Request
0-0	20113	0/751/3794	_	7.48	606	6	0.0	0.49	2.95	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
1-0	27522	0/640/3674	_	12.28	627	7	0.0	0.49	3.04	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1
2-0	20194	0/735/3776	_	7.80	627	6	0.0	0.42	4.36	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
3-0	27523	0/647/3685	_	44.08	442	162540	0.0	20.14	21.82	xx.yyy.yy.108	myserver.a.b.c	REPORT /my.path/!svn/blablabla/default HTTP/1.1
4-0	3076	0/1391/3425	W	14.89	0	0	0.0	0.90	2.42	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.
5-0	10815	0/397/3735	_	4.42	627	18	0.0	0.24	3.12	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/bblablabla/trunk H
6-0	27524	0/643/3682	_	13.14	606	33	0.0	1.68	3.58	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/vcc/default HTTP/1.1
7-0	28908	0/633/3666	_	7.06	627	9	0.0	0.42	5.62	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/vcc/default HTTP/1.1
8-0	27525	0/640/3673	_	13.35	627	9	0.0	4.97	7.44	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
9-0	27527	0/639/3675	_	27.84	627	11	0.0	2.62	5.38	xx.yyy.yy.114	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
10-0	27528	0/638/3677	_	8.09	606	3	0.0	0.60	7.14	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla HTTP/1.1
11-0	16355	0/301/3469	_	3.08	627	6	0.0	0.18	4.83	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla HTTP/1.1
12-0	2761	0/989/3282	_	10.41	627	6	0.0	0.67	2.72	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
13-0	29771	0/632/3667	_	6.51	627	16	0.0	0.38	3.40	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk HT
14-0	29928	0/586/3621	_	8.50	627	5	0.0	0.49	10.14	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla HTTP/1.1
15-0	29772	0/629/3666	_	6.77	606	8	0.0	0.37	3.15	xx.yyy.yy.108	myserver.a.b.c	REPORT /my.path/!svn/blablabla HTTP/1.1
16-0	27529	0/642/3682	_	6.56	627	6	0.0	0.36	3.67	xx.yyy.yy.114	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.1
17-0	27530	0/631/3672	_	11.19	606	6	0.0	0.43	3.80	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.1
18-0	27827	0/641/3678	_	6.79	615	5	0.0	0.38	2.20	xx.yyy.yy.140	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
19-0	28230	0/640/3098	_	7.22	627	6	0.0	0.39	1.72	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
20-0	28302	0/639/3679	_	7.11	606	6	0.0	0.48	4.59	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
21-0	28231	0/639/3675	W	8.22	0	0	0.0	0.60	4.22	xx.yyy.yy.5	myserver.a.b.c	GET /my.path/ HTTP/1.1
22-0	3077	0/1373/2899	_	26.72	627	17	0.0	3.16	10.28	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk
23-0	20679	0/235/3408	_	2.49	606	15	0.0	0.13	2.20	xx.yyy.yy.108	myserver.a.b.c	REPORT /my.path/!svn/blablabla/default HTTP/1.1
24-0	3078	0/1386/2905	_	16.83	627	14	0.0	2.16	3.08	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablablat/trunk
25-0	28297	0/616/2131	_	7.94	627	11	0.0	0.59	11.05	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
26-0	28299	0/634/3674	_	8.10	627	15	0.0	0.53	4.19	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk
27-0	28300	0/641/2157	_	6.58	606	5	0.0	0.34	1.12	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla HTTP/1.1
28-0	28303	0/642/2159	_	6.82	615	5	0.0	0.43	1.61	xx.yyy.yy.140	myserver.a.b.c	PROPFIND /my.path HTTP/1.1
29-0	28304	0/639/3673	_	7.29	615	10	0.0	0.50	7.08	xx.yyy.yy.140	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla HTTP/1.1
30-0	28305	0/633/2149	W	7.62	0	0	0.0	0.44	3.47	xx.yyy.yy.119	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
31-0	29929	0/585/3619	W	6.31	0	0	0.0	0.41	4.00	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.1
32-0	29930	0/592/3627	W	7.45	0	0	0.0	0.55	2.51	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.
33-0	28910	0/635/3673	W	8.87	0	0	0.0	0.38	2.44	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.1
34-0	28696	0/641/3676	_	6.57	606	98	0.0	0.38	10.64	xx.yyy.yy.108	myserver.a.b.c	REPORT /my.path/!svn/blablabla/default HTTP/1.1
35-0	6557	0/481/3517	W	4.98	0	0	0.0	0.33	8.89	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk H
36-0	28700	0/633/3671	_	7.61	627	6	0.0	0.38	2.22	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
37-0	29931	0/588/3621	_	6.22	627	14	0.0	0.36	13.57	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/testsSecurity
38-0	28697	0/626/3661	_	9.76	606	71	0.0	0.86	3.63	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
39-0	29766	0/628/3662	W	6.74	0	0	0.0	0.42	4.28	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/trunk/ HTTP/1.1
40-0	29932	0/586/3622	_	7.25	606	4	0.0	0.40	2.32	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
41-0	13813	0/366/3616	_	3.58	606	6	0.0	0.22	5.97	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
42-0	9	0/570/3606	_	5.83	627	13	0.0	0.37	21.85	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla HTTP/1.1
43-0	29770	0/634/3668	_	7.88	627	11	0.0	0.51	4.62	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
44-0	29916	0/614/3651	_	8.82	627	16	0.0	2.77	4.50	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
45-0	29905	0/631/3670	_	6.87	627	8	0.0	0.37	6.40	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
46-0	29906	0/623/3658	W	7.33	0	0	0.0	0.42	5.68	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1
47-0	29907	0/627/2494	W	7.11	0	0	0.0	0.51	1.57	xx.yyy.yy.243	myserver.a.b.c	GET /server-status HTTP/1.1
48-0	29917	0/615/3648	W	6.73	0	0	0.0	1.35	4.79	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1
49-0	29918	0/607/3640	_	9.02	606	6	0.0	0.79	3.13	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
50-0	29919	0/614/3648	_	6.30	627	17	0.0	0.35	2.08	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/trunk HTTP/
51-0	29933	0/587/3620	W	8.42	0	0	0.0	0.41	2.94	xx.yyy.yy.5	myserver.a.b.c	GET /my.path/trunk/ HTTP/1.1
52-0	29934	0/583/3618	_	7.54	606	6	0.0	0.47	3.41	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /blablabla/trunk HTTP/1.1
53-0	29935	0/593/3630	_	7.32	615	5	0.0	0.64	3.50	xx.yyy.yy.140	myserver.a.b.c	PROPFIND blablabla HTTP/1.1
54-0	6566	0/467/3504	W	5.79	0	0	0.0	0.28	2.77	xx.yyy.yy.140	myserver.a.b.c	PROPFIND /blablabla HTTP/1.1
55-0	6567	0/473/3508	_	4.58	606	6	0.0	0.27	2.09	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
56-0	27600	0/132/1651	_	1.71	627	9	0.0	0.14	7.04	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
57-0	27616	0/63/1602	_	29.34	615	5	0.0	19.80	22.77	xx.yyy.yy.140	myserver.a.b.c	PROPFIND /my.path HTTP/1.1
58-0	27617	0/118/1288	_	1.65	627	6	0.0	0.08	0.86	xx.yyy.yy.114	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1.1
59-0	27618	0/101/1277	W	0.96	0	0	0.0	0.07	3.19	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.1
60-0	27619	0/101/3137	_	1.22	627	5	0.0	0.08	2.15	xx.yyy.yy.114	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
61-0	27621	0/101/3133	_	1.01	615	5	0.0	0.06	1.77	xx.yyy.yy.140	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
62-0	27623	0/99/3133	_	1.44	627	18	0.0	0.09	1.81	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla
63-0	-	0/0/1513	.	19.93	10227	11	0.0	0.00	1.34	xx.yyy.yy.108	myserver.a.b.c	OPTIONS /my.path/blablabla/trunk HTTP/1.1
64-0	-	0/0/1513	.	16.68	10235	6	0.0	0.00	1.06	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
65-0	-	0/0/1511	.	51.59	10167	9	0.0	0.00	20.72	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/4389 HTTP/1.1
66-0	-	0/0/1520	.	17.51	4947	7	0.0	0.00	1.19	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/blablabla/trunk HTTP/1
67-0	-	0/0/1520	.	17.32	4998	7	0.0	0.00	1.32	xx.yyy.yy.30	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
68-0	-	0/0/1520	.	29.50	4995	10	0.0	0.00	2.92	xx.yyy.yy.30	myserver.a.b.c	REPORT /my.path/!svn/blablabla/branches HTTP/1.1
69-0	-	0/0/784	.	8.89	12044	0	0.0	0.00	0.50	127.0.0.1	myserver.a.b.c	OPTIONS * HTTP/1.0
70-0	-	0/0/1524	.	17.60	4849	46	0.0	0.00	2.01	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
71-0	-	0/0/1520	.	28.70	4850	18	0.0	0.00	2.42	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/
72-0	-	0/0/1520	.	15.69	5015	7	0.0	0.00	0.89	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
73-0	-	0/0/1520	.	31.96	4955	19	0.0	0.00	8.35	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla
74-0	-	0/0/1521	.	18.82	4998	4	0.0	0.00	0.95	xx.yyy.yy.30	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
75-0	-	0/0/1520	.	16.71	5015	8	0.0	0.00	0.96	xx.yyy.yy.108	myserver.a.b.c	PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

------------------------------------------------------------------------

Here is the extract from our httpd.conf file

ServerRoot "/opt/xxxxxx/xxx/xxxxxxxxxxxxxxxxxxxxxxx"
LockFile accept.lock
PidFile apache2.pid

Listen 0.0.0.0:20001
LockFile accept.lock
PidFile apache2.pid
Timeout 300
KeepAlive Off # was On before
MaxKeepAliveRequests 1536
KeepAliveTimeout 2

<IfModule mpm_prefork_module>
  StartServers                  32
  MinSpareServers         48
  MaxSpareServers         64
  ServerLimit                    512
  MaxClients                    96
  MaxRequestsPerChild   1536
</IfModule>

.........

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/opt/blablabla/cgi-bin/"
</IfModule>

<Directory "/opt/blablabla/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

# Main server Admin SVN
ServerName scm.xyz.com
DocumentRoot "/opt/blablabla/htdocs"

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "/opt/blablabla/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

# PHP5 :
Include conf/extra/httpd-php.conf

# Repository SVN
Listen 0.0.0.0:20000
<VirtualHost *:20000>
    Include conf/extra/httpd-svn.conf
</VirtualHost>

Any help appreciated.

/Nourrédine

--

Nourredine Rouibah
nourredine.rouibah@xxxxxxxxx