Thought to post some more debugging regarding this problem I have.
This time I had help from one of our Unix engineers with the debugging
and you can see he used mdb in this case:
bash-2.05$ mdb core.httpd.28499.wenpweb1.500.500.1253659663
mdb: warning: failed to infer pathname to executable; symbol table
will not be available
Loading modules: [ libc.so.1 libthread.so.1 ld.so.1 ]
> $C (display call stack)
fbef95d8 libapr-1.so.0`apr_palloc+4(44617465, 7, 70740020, 7efefeff,
81010100, ff00)
fbef9648 libapr-1.so.0`apr_pstrdup+0x20(44617465, 27a758, 0, 13f75a4, 0, 27a75a)
fbef96b8 libapr-1.so.0`apr_table_add+0xa8(13f75a0, 27a758, 27a760, 5,
fffc0000, d)
fbef9728 mod_mem_cache.so`deep_table_copy+0x60(13f75a0, 0, 5, 2ba944, 38, 1)
fbef9798 mod_mem_cache.so`store_headers+0x1c(ecba0, 2797f8, ecba8,
26b900, ecba8, fec522d4)
(snip)
> apr_palloc::dis (disassemble apr_palloc)
libapr-1.so.0`apr_palloc: save %sp, -0x70, %sp
libapr-1.so.0`apr_palloc+4: ld [%i0 + 0x2c], %l0
libapr-1.so.0`apr_palloc+8: mov %i0, %l1
(snip)
> $r (display registers)
(snip)
%o0 = 0x00008ca0 %i0 = 0x44617465
(snip)
%i0 (the first parameter being passed to a function) is an invalid
pointer. Looking at it, it actually appears to be part of a text
string:
0x44 = D
0x61 = a
0x74 = t
0x65 = e
Putting all this together, apr_table_add is dereferencing a string and
passing the contents of it to apr_pstrdup, instead of passing a
pointer to the memory pool (which is what apr_pstrdup expects)
Most likely something in apr_table_add is getting over-written.
Other cores show similar behaviour:
hmurn@wenpweb1# mdb core.httpd.28752.wenpweb1.500.500.1254735017
> $C
fcff95d8 libapr-1.so.0`apr_palloc+4(436f6e74 ('Cont'), 7, 70740020,
7efefeff, 81010100, ff00)
bash-2.05# mdb core.httpd.12737.wenpweb1.500.500.1254699635
> $C
fccf96b8 libapr-1.so.0`apr_table_add+0xb4(56f5c10, 527d0f8, 5281740, 5, 1, 6)
> apr_table_add+0xb4::dis
libapr-1.so.0`apr_table_add+0xb4: st %o0, [%l0]
> $r
%g0 = 0x00000000 %l0 = 0x68747470 ('http')
So it really looks like a bug in the apache apr so the patch that Nick
suggested makes more sense for me now.
Nik, do you know if this is the same issue that the patch is suppose
to fix? Can you please point me to the patch release documentation so
I can read some more about it?
Thanks for your help.
Igor
On 9/24/09, Igor Cicimov <icicimov@xxxxxxxxx> wrote:
> Ah forgot about the tool ... used pstack to debug the core dumps since it
> is
> a production server and don't have gdb available.
>
> Cheers,
>
> Igor
>
> On Thu, Sep 24, 2009 at 12:37 PM, Igor Cicimov <icicimov@xxxxxxxxx> wrote:
>
>> Hi Nick,
>>
>> First thanks for your reply much appreciate it.
>>
>> Yes, the server is reverse proxy only and from what I could see from
>> couple
>> of other core dumps they all look the same. First I thought it might be
>> the
>> proxy_html module causing this but now I'm not that sure and might be
>> something more serious.
>>
>> I'll have a look in the patch and see if it is applicable to me.
>>
>> Thanks,
>>
>> Igor
>>
>>
>> On Wed, Sep 23, 2009 at 6:55 PM, Nick Kew <nick@xxxxxxxxxxxx> wrote:
>>
>>>
>>> On 23 Sep 2009, at 07:32, Igor Cicimov wrote:
>>>
>>> Hi all,
>>>>
>>>> I get the following core dumps from the apache:
>>>> [chop]
>>>>
>>>> It's apache 2.2.12 running as reverse proxy on Solaris 9 box with
>>>> mpm_worker module. Looks like the threads get in some conditional wait
>>>> state
>>>> and the process becomes zombie. I can see individual child processes
>>>> dieing
>>>> in the log file with segmentation fault.
>>>>
>>>
>>> What tool produced that dump? Do you have gdb available?
>>> (a gdb backtrace means more to me than what you show).
>>>
>>> You mention it's a reverse proxy, and the first thread you show is
>>> a proxy request. Are the crashes always associated with that,
>>> so we could look there for a cause? Or is that a meaningless
>>> question because the server does nothing else?
>>>
>>> You might also apply Jeff's APR patch at
>>> https://issues.apache.org/bugzilla/attachment.cgi?id=24161
>>> and see if that helps.
>>>
>>> --
>>> Nick Kew
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>>> " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
>>> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>>>
>>>
>>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|