Just another oddity to add to the issue.Overnight several more hung threads appeared and the load on the system had jumped into the mid 20's.After killing these the load did not drop. Looking at the list of running processes I found httpd's running,spawned from the original root httpd process that *were not even displayed* in the scoreboard!! After killing these hidden zombies off the load dropped again.
So now I have to catch and kill two types: Zombies on the scoreboard and hidden zombies.And this is cute. Some times the zombies hang around so long that when the system gets back to creating a new process for slot #1, if the zombie was originally in that slot it is displayed their along with it's brothers for the new process:
e.g. Note process 19597 below1-0166310/33/1320_ 131.22202255280.01.6035.79 10.172.91.217newyahoo.oak.sap.corp:80NULL 1-0166310/18/1087_ 105.88340736980.00.6926.65 10.172.240.113www-dse.oak.sap.corp:80GET /cgi-bin/websql/websql.dir/QTS/bugsheetcont.hts?bugid=74133 1-0166310/11/1178_ 76.49589542980.00.5634.78 10.172.91.92newyahoo.oak.sap.corp:80NULL 1-0166310/32/1295_ 92.17425417130.04.0342.07 10.172.240.113newyahoo.oak.sap.corp:80NULL 1-0195970/26/1319W 35.552441700.00.5437.10 10.172.248.87www-rev.oak.sap.corp:80GET /cgi-bin/rev.cgi?action="" HTTP/1.1 1-0166310/12/1427_ 18.41794100.00.14238.52 10.172.240.113newyahoo.oak.sap.corp:80NULL 1-0166310/27/1442_ 30.67719695430.00.7835.07 10.172.85.9newyahoo.oak.sap.corp:80NULL 1-0166310/19/784_ 10.70940630.00.4520.95 10.172.246.203newyahoo.oak.sap.corp:80NULL 1-0166310/8/1034_ 2.86103144630.00.0124.04 10.172.90.155newyahoo.oak.sap.corp:80NULL 2-0-0/0/99. 58.943145013820.00.002.15 10.136.66.135newyahoo.oak.sap.corp:80NULL 2-0-0/0/82. 2181.923144824390.00.001.48 10.162.65.165www-dse.oak.sap.corp:80POST /cgi-bin/websql/websql.dir/QTS/bugsescalated.pl?product=AN 2-0-0/0/162. 2027.12314509350.00.003.36 10.50.3.99newyahoo.oak.sap.corp:80NULL 2-0-0/0/576. 1704.40314504100.00.0013.38 10.172.240.113newyahoo.oak.sap.corp:80NULL 2-0-0/0/928. 1295.363145029750.00.0024.38 10.50.17.221newyahoo.oak.sap.corp:80NULL 2-0-0/0/852. 1798.52314503810.00.0020.72 10.162.65.165newyahoo.oak.sap.corp:80NULL 2-0-0/0/1084. 551.293145022210.00.0026.52 10.176.138.162newyahoo.oak.sap.corp:80NULL 2-0-0/0/1180. 385.833145019630.00.0034.31 10.162.65.197newyahoo.oak.sap.corp:80NULL 2-0-0/0/50. 50.713145000.00.001.62 10.58.181.166www-rev.oak.sap.corp:80GET /cgi-bin/rev.cgi?action="" HTTP/1.1 2-0137610/12/1078W 58.803489600.00.1031.67 10.172.107.38www-rev.oak.sap.corp:80POST /cgi-bin/rev.cgi HTTP/1.1 2-0-0/0/1075. 1061.5331450790.00.0031.65 10.172.90.88newyahoo.oak.sap.corp:80GET /server-status HTTP/1.1 2-0-0/0/1362. 46.803145080.00.0039.72 10.172.107.38www-rev.oak.sap.corp:80POST /cgi-bin/rev.cgi HTTP/1.1 2-0-0/0/1142. 56.693145011490.00.0035.22 10.172.240.113newyahoo.oak.sap.corp:80NULSlot #2 currently not being used (still has zombie)MJMjOn Tuesday, June 16, 2015 5:42 PM, Mark Jacquet <mark_jacquet@xxxxxxxxx.INVALID> wrote:
Upgrade as in Apache upgrade or Solaris 5.10 patch upgrad? :)Apache is all new of course 2.4.12 with the latest add on sources (apr, pcre, etc)The bad news is the OS is not at all up to date. And for reasons I have no control over, I cannot patch.So if this is an OS issue then ......I seem to be running with the Sun Native LDAP SDK. Would building against different LDAP source help? (Open LDAP)?Long term plan -> moving all Apache servers to LinuxMjOn Tuesday, June 16, 2015 5:31 PM, Eric Covener <covener@xxxxxxxxx> wrote:
On Tue, Jun 16, 2015 at 8:23 PM, Mark Jacquet
<mark_jacquet@xxxxxxxxx.invalid> wrote:
> So do you think this hang is related to the native LDAP lib code?
It is possible but IMO not very likely. It has to corrutp memory just
enough to put a looping structure in apr_rmm. What's your upgrade
history like?
--
Eric Covener
covener@xxxxxxxxx
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|