Apache httpd 2.2.12 hangs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hello,

Our httpd Apache server keeps hanging every day or many times a day depending on the activity.
Apache is the frontend of our Subversion 1.6.5 server.
We have about a 100 client developers accessing SVN thru Apache and a few Hudson continuous integration servers accessing SVN as well and running many automatic SVN accesses 24/7.
SVN is working well because when the problem happens, I can isssue local SVN commands outside of httpd without any problem.

Our Apache is listening to 2 different ports 20000 and 20001.
Port 20000 is the standard SVN port and port 20001 is used by the SVN PHP Admin tools. (svnadmin and websvn)

When the httpd Apache hangs it is always on port 20000.
All SVN requests are stuck for some reasons.

When the httpd hangs on port 20000, if we issue an http request using svnadmin on port 20001 then the server is unblocked for a short moment sometime few minutes after hanging again.
Then if we re-issue a request on port 20001 it will again release port 20000 requests for a while.
The only clean solution is to restart the httpd server.
Then it will eventualy run for a few hours or a day or two and then hang again.

We turned keepalive to OFF to test if we were hiting socket release issues but the problem just happened with KEEPALIVE set to OFF.
Our server is a large solaris machine with plenty of memory and swap.
Solaris : SunOS MyMachine 5.10 Generic_147440-24 sun4v sparc sun4v
Apache: Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2 SVN/1.6.5 PHP/5.3.1

After the apache is hung here is the status-server request:

Apache Server Status for scm.xyz.com
Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2 SVN/1.6.5 PHP/5.3.1
Server Built: Jan 13 2010 15:49:24
Current Time: Friday, 16-Aug-2013 15:21:52 CEST
Restart Time: Friday, 16-Aug-2013 03:10:02 CEST
Parent Server Generation: 0
Server uptime: 12 hours 11 minutes 50 seconds

Total accesses: 228561 - Total Traffic: 369.5 MB
CPU Usage: u637.01 s183.88 cu1.4 cs0 - 1.87% CPU load
5.21 requests/sec - 8.6 kB/second - 1694 B/request
8 requests currently being processed, 55 idle workers

_______________________________WWW_W___W______WWR______________.
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv

PID

Acc

M

CPU

SS

Req

Conn

Child

Slot

Client

VHost

Request

0-0

20113

0/751/3794

_

7.48

606

6

0.0

0.49

2.95

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

1-0

27522

0/640/3674

_

12.28

627

7

0.0

0.49

3.04

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1

2-0

20194

0/735/3776

_

7.80

627

6

0.0

0.42

4.36

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

3-0

27523

0/647/3685

_

44.08

442

162540

0.0

20.14

21.82

xx.yyy.yy.108

myserver.a.b.c

REPORT /my.path/!svn/blablabla/default HTTP/1.1

4-0

3076

0/1391/3425

W

14.89

0

0

0.0

0.90

2.42

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.

5-0

10815

0/397/3735

_

4.42

627

18

0.0

0.24

3.12

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/bblablabla/trunk H

6-0

27524

0/643/3682

_

13.14

606

33

0.0

1.68

3.58

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/vcc/default HTTP/1.1

7-0

28908

0/633/3666

_

7.06

627

9

0.0

0.42

5.62

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/vcc/default HTTP/1.1

8-0

27525

0/640/3673

_

13.35

627

9

0.0

4.97

7.44

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

9-0

27527

0/639/3675

_

27.84

627

11

0.0

2.62

5.38

xx.yyy.yy.114

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1

10-0

27528

0/638/3677

_

8.09

606

3

0.0

0.60

7.14

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla HTTP/1.1

11-0

16355

0/301/3469

_

3.08

627

6

0.0

0.18

4.83

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla HTTP/1.1

12-0

2761

0/989/3282

_

10.41

627

6

0.0

0.67

2.72

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

13-0

29771

0/632/3667

_

6.51

627

16

0.0

0.38

3.40

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk HT

14-0

29928

0/586/3621

_

8.50

627

5

0.0

0.49

10.14

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla HTTP/1.1

15-0

29772

0/629/3666

_

6.77

606

8

0.0

0.37

3.15

xx.yyy.yy.108

myserver.a.b.c

REPORT /my.path/!svn/blablabla HTTP/1.1

16-0

27529

0/642/3682

_

6.56

627

6

0.0

0.36

3.67

xx.yyy.yy.114

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.1

17-0

27530

0/631/3672

_

11.19

606

6

0.0

0.43

3.80

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.1

18-0

27827

0/641/3678

_

6.79

615

5

0.0

0.38

2.20

xx.yyy.yy.140

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

19-0

28230

0/640/3098

_

7.22

627

6

0.0

0.39

1.72

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

20-0

28302

0/639/3679

_

7.11

606

6

0.0

0.48

4.59

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

21-0

28231

0/639/3675

W

8.22

0

0

0.0

0.60

4.22

xx.yyy.yy.5

myserver.a.b.c

GET /my.path/ HTTP/1.1

22-0

3077

0/1373/2899

_

26.72

627

17

0.0

3.16

10.28

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk

23-0

20679

0/235/3408

_

2.49

606

15

0.0

0.13

2.20

xx.yyy.yy.108

myserver.a.b.c

REPORT /my.path/!svn/blablabla/default HTTP/1.1

24-0

3078

0/1386/2905

_

16.83

627

14

0.0

2.16

3.08

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablablat/trunk

25-0

28297

0/616/2131

_

7.94

627

11

0.0

0.59

11.05

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

26-0

28299

0/634/3674

_

8.10

627

15

0.0

0.53

4.19

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk

27-0

28300

0/641/2157

_

6.58

606

5

0.0

0.34

1.12

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla HTTP/1.1

28-0

28303

0/642/2159

_

6.82

615

5

0.0

0.43

1.61

xx.yyy.yy.140

myserver.a.b.c

PROPFIND /my.path HTTP/1.1

29-0

28304

0/639/3673

_

7.29

615

10

0.0

0.50

7.08

xx.yyy.yy.140

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla HTTP/1.1

30-0

28305

0/633/2149

W

7.62

0

0

0.0

0.44

3.47

xx.yyy.yy.119

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

31-0

29929

0/585/3619

W

6.31

0

0

0.0

0.41

4.00

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.1

32-0

29930

0/592/3627

W

7.45

0

0

0.0

0.55

2.51

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.

33-0

28910

0/635/3673

W

8.87

0

0

0.0

0.38

2.44

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.1

34-0

28696

0/641/3676

_

6.57

606

98

0.0

0.38

10.64

xx.yyy.yy.108

myserver.a.b.c

REPORT /my.path/!svn/blablabla/default HTTP/1.1

35-0

6557

0/481/3517

W

4.98

0

0

0.0

0.33

8.89

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk H

36-0

28700

0/633/3671

_

7.61

627

6

0.0

0.38

2.22

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

37-0

29931

0/588/3621

_

6.22

627

14

0.0

0.36

13.57

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/testsSecurity

38-0

28697

0/626/3661

_

9.76

606

71

0.0

0.86

3.63

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

39-0

29766

0/628/3662

W

6.74

0

0

0.0

0.42

4.28

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/trunk/ HTTP/1.1

40-0

29932

0/586/3622

_

7.25

606

4

0.0

0.40

2.32

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

41-0

13813

0/366/3616

_

3.58

606

6

0.0

0.22

5.97

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

42-0

9

0/570/3606

_

5.83

627

13

0.0

0.37

21.85

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla HTTP/1.1

43-0

29770

0/634/3668

_

7.88

627

11

0.0

0.51

4.62

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

44-0

29916

0/614/3651

_

8.82

627

16

0.0

2.77

4.50

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1

45-0

29905

0/631/3670

_

6.87

627

8

0.0

0.37

6.40

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

46-0

29906

0/623/3658

W

7.33

0

0

0.0

0.42

5.68

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1

47-0

29907

0/627/2494

W

7.11

0

0

0.0

0.51

1.57

xx.yyy.yy.243

myserver.a.b.c

GET /server-status HTTP/1.1

48-0

29917

0/615/3648

W

6.73

0

0

0.0

1.35

4.79

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1

49-0

29918

0/607/3640

_

9.02

606

6

0.0

0.79

3.13

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1

50-0

29919

0/614/3648

_

6.30

627

17

0.0

0.35

2.08

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/trunk HTTP/

51-0

29933

0/587/3620

W

8.42

0

0

0.0

0.41

2.94

xx.yyy.yy.5

myserver.a.b.c

GET /my.path/trunk/ HTTP/1.1

52-0

29934

0/583/3618

_

7.54

606

6

0.0

0.47

3.41

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /blablabla/trunk HTTP/1.1

53-0

29935

0/593/3630

_

7.32

615

5

0.0

0.64

3.50

xx.yyy.yy.140

myserver.a.b.c

PROPFIND blablabla HTTP/1.1

54-0

6566

0/467/3504

W

5.79

0

0

0.0

0.28

2.77

xx.yyy.yy.140

myserver.a.b.c

PROPFIND /blablabla HTTP/1.1

55-0

6567

0/473/3508

_

4.58

606

6

0.0

0.27

2.09

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

56-0

27600

0/132/1651

_

1.71

627

9

0.0

0.14

7.04

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

57-0

27616

0/63/1602

_

29.34

615

5

0.0

19.80

22.77

xx.yyy.yy.140

myserver.a.b.c

PROPFIND /my.path HTTP/1.1

58-0

27617

0/118/1288

_

1.65

627

6

0.0

0.08

0.86

xx.yyy.yy.114

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1.1

59-0

27618

0/101/1277

W

0.96

0

0

0.0

0.07

3.19

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.1

60-0

27619

0/101/3137

_

1.22

627

5

0.0

0.08

2.15

xx.yyy.yy.114

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

61-0

27621

0/101/3133

_

1.01

615

5

0.0

0.06

1.77

xx.yyy.yy.140

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

62-0

27623

0/99/3133

_

1.44

627

18

0.0

0.09

1.81

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla

63-0

-

0/0/1513

.

19.93

10227

11

0.0

0.00

1.34

xx.yyy.yy.108

myserver.a.b.c

OPTIONS /my.path/blablabla/trunk HTTP/1.1

64-0

-

0/0/1513

.

16.68

10235

6

0.0

0.00

1.06

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

65-0

-

0/0/1511

.

51.59

10167

9

0.0

0.00

20.72

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/4389 HTTP/1.1

66-0

-

0/0/1520

.

17.51

4947

7

0.0

0.00

1.19

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/blablabla/trunk HTTP/1

67-0

-

0/0/1520

.

17.32

4998

7

0.0

0.00

1.32

xx.yyy.yy.30

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

68-0

-

0/0/1520

.

29.50

4995

10

0.0

0.00

2.92

xx.yyy.yy.30

myserver.a.b.c

REPORT /my.path/!svn/blablabla/branches HTTP/1.1

69-0

-

0/0/784

.

8.89

12044

0

0.0

0.00

0.50

127.0.0.1

myserver.a.b.c

OPTIONS * HTTP/1.0

70-0

-

0/0/1524

.

17.60

4849

46

0.0

0.00

2.01

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

71-0

-

0/0/1520

.

28.70

4850

18

0.0

0.00

2.42

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/

72-0

-

0/0/1520

.

15.69

5015

7

0.0

0.00

0.89

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

73-0

-

0/0/1520

.

31.96

4955

19

0.0

0.00

8.35

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla

74-0

-

0/0/1521

.

18.82

4998

4

0.0

0.00

0.95

xx.yyy.yy.30

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

75-0

-

0/0/1520

.

16.71

5015

8

0.0

0.00

0.96

xx.yyy.yy.108

myserver.a.b.c

PROPFIND /my.path/!svn/blablabla/default HTTP/1.1

------------------------------------------------------------------------

Here is the extract from our httpd.conf file

ServerRoot "/opt/xxxxxx/xxx/xxxxxxxxxxxxxxxxxxxxxxx"
LockFile accept.lock
PidFile apache2.pid

Listen 0.0.0.0:20001
LockFile accept.lock
PidFile apache2.pid
Timeout 300
KeepAlive Off # was On before
MaxKeepAliveRequests 1536
KeepAliveTimeout 2

<IfModule mpm_prefork_module>
  StartServers                  32
  MinSpareServers         48
  MaxSpareServers         64
  ServerLimit                    512
  MaxClients                    96
  MaxRequestsPerChild   1536
</IfModule>

.........

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/opt/blablabla/cgi-bin/"
</IfModule>

<Directory "/opt/blablabla/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

# Main server Admin SVN
ServerName scm.xyz.com
DocumentRoot "/opt/blablabla/htdocs"

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "/opt/blablabla/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

# PHP5 :
Include conf/extra/httpd-php.conf

# Repository SVN
Listen 0.0.0.0:20000
<VirtualHost *:20000>
    Include conf/extra/httpd-svn.conf
</VirtualHost>

Any help appreciated.

/Nourrédine
[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux