We recently started experiencing a problem on our server (httpd 2.2.19, PHP 5.3.6) where all of the apache child processes hang in PHP code and no longer process requests.
I am running Apache as prefork on Solaris 10 with 256 children, and found that within a few minutes all 256 httpd child processes are stuck, and it's always in the same place within php_session_start in a php_flock call. Stack trace is shown below.
The file it is trying to flock is /var/tmp/sess_2e39aabaf226819b67f29da477892e91.
All 256 httpd child processes show the exact same stack trace. When it occurs it eventually (after a long time does timeout) and recover - but it may take over 1 hour. We have to stop and restart the server before that.
pstack 6802
6802: /usr/local/apache/bin/httpd -d /usr/local/apache -f /usr/local/apache/
feccb880 fcntl (12, 7, ffbfe600)
fecbb514 fcntl (12, 1, ff172a00, 7d054, fe7845c4, fed303a8) + 84
fe6f2d88 php_flock (12, 2, 0, 1c80e8, 0, 0) + 58
fe624e38 ps_files_open (1c9af0, 1c80e8, ffffffff, fea76238, 1c6738, 100) + 224
fe625160 ps_read_files (1c9af0, 1c80e8, ffbfeca0, ffbfec9c, 4, fe625148) + 18
fe622984 php_session_start (18, fffb8bf0, ffffffff, b800, b400, b800) + bd0
fe623e34 zif_session_start (0, 1cfeb0, 0, 0, 0, 333f18) + 10
fe7ca6b8 zend_do_fcall_common_helper_SPEC (333460, b400, feb25e18, 33bd78, 0, 0) + 76c
fe7c99e0 execute (feb25e18, 1c8000, 2f, ffbfeed4, 4, feb24d00) + 210
fe7a6494 zend_execute_scripts (8, 1, 3, ffbfef40, feb25e18, ffbff4f8) + 12c
fe74c0c4 php_execute_script (0, feab1f20, 6, 0, 70687000, 0) + 224
fe83a470 php_handler (32d470, 0, 32d470, c2c00, c3ee0, 9) + 2d0
000412c8 ap_run_handler (32d470, 3b3b3b3b, 70687000, 80808080, ff00, 80808080)
+ 3c
0004175c ap_invoke_handler (1668c0, 97400, 32d470, ffbff68c, fddf0028, 0) + c0
0006c890 ap_process_request (32d470, 0, 4, 32d470, 0, 0) + 160
00069964 ap_process_http_connection (3276f0, 327458, 327458, b6, c4740, e3dc0)+ 10c
00047a60 ap_run_process_connection (3276f0, 327458, 327458, b6, 3254a0, 32b428) + 3c
00087740 child_main (0, 1, c6800, c6c00, 11177, c6800) + 434
000879a4 make_child (87000, b6, 6, 0, d7300, c6800) + ec
000883ac ap_mpm_run (c6800, c4400, aa, c6800, c6c00, c6800) + 91c
0002e2bc main (d17a8, c1800, c3c00, c3c00, cf7a0, 0) + 784
0002d668 _start (0, 0, 0, 0, 0, 0) + 5c
# pfiles 6802
6802: /usr/local/apache/bin/httpd -d /usr/local/apache -f /usr/local/apache/
Current rlimit: 65536 file descriptors
0: S_IFCHR mode:0666 dev:340,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDONLY
/devices/pseudo/mm@0:null
1: S_IFCHR mode:0666 dev:340,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_WRONLY|O_CREAT|O_TRUNC
/devices/pseudo/mm@0:null
2: S_IFREG mode:0644 dev:85,0 ino:270102 uid:60001 gid:60001 size:3692960
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/usr/local/apache/logs/error_log
3: S_IFSOCK mode:0666 dev:347,0 ino:32405 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 80
4: S_IFDOOR mode:0444 dev:349,0 ino:55 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[170]
/var/run/name_service_door
5: S_IFCHR mode:0620 dev:340,0 ino:12582920 uid:0 gid:7 rdev:24,2
O_RDWR
/devices/pseudo/pts@0:2
6: S_IFSOCK mode:0666 dev:347,0 ino:52766 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0
.0.192.0)
sockname: AF_INET 0.0.0.0 port: 443
7: S_IFIFO mode:0000 dev:338,0 ino:2827255 uid:0 gid:1 size:0
O_RDWR|O_NONBLOCK
8: S_IFIFO mode:0000 dev:338,0 ino:2827255 uid:0 gid:1 size:0
O_RDWR
9: S_IFREG mode:0644 dev:85,0 ino:270103 uid:60001 gid:60001 size:601185983
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/usr/local/apache/logs/access_log
10: S_IFREG mode:0644 dev:85,0 ino:270103 uid:60001 gid:60001 size:601185983
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/usr/local/apache/logs/access_log
11: S_IFREG mode:0644 dev:85,0 ino:270104 uid:60001 gid:60001 size:2151
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/usr/local/apache/logs/ssl_request_log
12: S_IFREG mode:0644 dev:85,0 ino:270797 uid:0 gid:1 size:0
O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE
13: S_IFCHR mode:0000 dev:340,0 ino:18026 uid:0 gid:0 rdev:21,622
O_WRONLY FD_CLOEXEC
/devices/pseudo/log@0:conslog
14: S_IFREG mode:0644 dev:85,0 ino:270798 uid:0 gid:1 size:0
O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE
15: S_IFPORT mode:0000 dev:350,0 uid:60001 gid:60001 size:0
16: S_IFSOCK mode:0666 dev:347,0 ino:19961 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49640),IP_NEXTHOP(0.0.193.232)
sockname: AF_INET 128.96.156.65 port: 80
peername: AF_INET 128.96.195.58 port: 1672
17: S_IFDIR mode:0755 dev:85,0 ino:2 uid:0 gid:0 size:1024
O_RDONLY
/
18: S_IFREG mode:0600 dev:85,0 ino:759973 uid:60001 gid:60001 size:1072
O_RDWR|O_CREAT
advisory write lock set by process 6706
/var/tmp/sess_2e39aabaf226819b67f29da477892e91
#
fuser -f sess_2e39aabaf226819b67f29da477892e91
sess_2e39aabaf226819b67f29da477892e91: 6885o 6884o 6883o 6882o 6
881o 6880o 6879o 6878o 6877o 6876o 6875o 6874o 6873o
6872o 6871o 6870o 6869o 6868o 6867o 6866o 6865o 6854o
6853o 6852o 6851o 6850o 6849o 6848o 6847o 6846o 6845o...
brief outline of my code is:
session_cache_limiter('nocache');
session_start();
store/retrieve some data from $_SESSION
ob_start();
code to format page to send to browser.
ob_end_flush();
session_write_close();
exit();
I appreciate any advice as to how to avoid this problem, or experiences with it for my trouble shooting.
thank you.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
