|
Sadly we don't have a core dump when the crash happened, so that we are not able to dump the registers..
The latest status - we changed the rgw thread number from 600 to 300, and we haven't seen the same crash since, but still it is hard to tell if that is related and how it is related..
Thanks,
Guang
> Date: Wed, 24 Jun 2015 17:21:04 -0400
> From: yehuda@xxxxxxxxxx
> To: yguang11@xxxxxxxxxxx
> CC: ceph-devel@xxxxxxxxxxxxxxx; ceph-users@xxxxxxxxxxxxxx
> Subject: Re: radosgw crash within libfcgi
>
>
>
> ----- Original Message -----
>> From: "GuangYang"
>> To: "Yehuda Sadeh-Weinraub"
>> Cc: ceph-devel@xxxxxxxxxxxxxxx, ceph-users@xxxxxxxxxxxxxx
>> Sent: Wednesday, June 24, 2015 2:12:23 PM
>> Subject: RE: radosgw crash within libfcgi
>>
>> ----------------------------------------
>>> Date: Wed, 24 Jun 2015 17:04:05 -0400
>>> From: yehuda@xxxxxxxxxx
>>> To: yguang11@xxxxxxxxxxx
>>> CC: ceph-devel@xxxxxxxxxxxxxxx; ceph-users@xxxxxxxxxxxxxx
>>> Subject: Re: radosgw crash within libfcgi
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "GuangYang"
>>>> To: "Yehuda Sadeh-Weinraub"
>>>> Cc: ceph-devel@xxxxxxxxxxxxxxx, ceph-users@xxxxxxxxxxxxxx
>>>> Sent: Wednesday, June 24, 2015 1:53:20 PM
>>>> Subject: RE: radosgw crash within libfcgi
>>>>
>>>> Thanks Yehuda for the response.
>>>>
>>>> We already patched libfcgi to use poll instead of select to overcome the
>>>> limitation.
>>>>
>>>> Thanks,
>>>> Guang
>>>>
>>>>
>>>> ----------------------------------------
>>>>> Date: Wed, 24 Jun 2015 14:40:25 -0400
>>>>> From: yehuda@xxxxxxxxxx
>>>>> To: yguang11@xxxxxxxxxxx
>>>>> CC: ceph-devel@xxxxxxxxxxxxxxx; ceph-users@xxxxxxxxxxxxxx
>>>>> Subject: Re: radosgw crash within libfcgi
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "GuangYang"
>>>>>> To: ceph-devel@xxxxxxxxxxxxxxx, ceph-users@xxxxxxxxxxxxxx,
>>>>>> yehuda@xxxxxxxxxx
>>>>>> Sent: Wednesday, June 24, 2015 10:09:58 AM
>>>>>> Subject: radosgw crash within libfcgi
>>>>>>
>>>>>> Hello Cephers,
>>>>>> Recently we have several radosgw daemon crashes with the same following
>>>>>> kernel log:
>>>>>>
>>>>>> Jun 23 14:17:38 xxx kernel: radosgw[68180]: segfault at f0 ip
>>>>>> 00007ffa069996f2 sp 00007ff55c432710 error 6 in
>>>
>>> error 6 is sigabrt, right? With invalid pointer I'd expect to get segfault.
>>> Is the pointer actually invalid?
>> With (ip - {address_load_the_sharded_library}) to get the instruction which
>> caused this crash, the objdump shows the crash happened at instruction 46f2
>> (see below), which was to assign '-1' to the CGX_Request::ipcFd to -1, but I
>> don't quite understand how/why it could crash there.
>>
>> 0000000000004690 :
>> 4690: 48 89 5c 24 f0 mov %rbx,-0x10(%rsp)
>> 4695: 48 89 6c 24 f8 mov %rbp,-0x8(%rsp)
>> 469a: 48 83 ec 18 sub $0x18,%rsp
>> 469e: 48 85 ff test %rdi,%rdi
>> 46a1: 48 89 fb mov %rdi,%rbx
>> 46a4: 89 f5 mov %esi,%ebp
>> 46a6: 74 28 je 46d0
>> 46a8: 48 8d 7f 08 lea 0x8(%rdi),%rdi
>> 46ac: e8 67 e3 ff ff callq 2a18
>> 46b1: 48 8d 7b 10 lea 0x10(%rbx),%rdi
>> 46b5: e8 5e e3 ff ff callq 2a18
>> 46ba: 48 8d 7b 18 lea 0x18(%rbx),%rdi
>> 46be: e8 55 e3 ff ff callq 2a18
>> 46c3: 48 8d 7b 28 lea 0x28(%rbx),%rdi
>> 46c7: e8 d4 f4 ff ff callq 3ba0
>> 46cc: 85 ed test %ebp,%ebp
>> 46ce: 75 10 jne 46e0
>> 46d0: 48 8b 5c 24 08 mov 0x8(%rsp),%rbx
>> 46d5: 48 8b 6c 24 10 mov 0x10(%rsp),%rbp
>> 46da: 48 83 c4 18 add $0x18,%rsp
>> 46de: c3 retq
>> 46df: 90 nop
>> 46e0: 31 f6 xor %esi,%esi
>> 46e2: 83 7b 4c 00 cmpl $0x0,0x4c(%rbx)
>> 46e6: 8b 7b 30 mov 0x30(%rbx),%edi
>> 46e9: 40 0f 94 c6 sete %sil
>> 46ed: e8 86 e6 ff ff callq 2d78
>> 46f2: c7 43 30 ff ff ff ff movl $0xffffffff,0x30(%rbx)
>
> info registers?
>
> Not too familiar with the specific message, but it could be that OS_IpcClose() aborts (not highly unlikely) and it only dumps the return address of the current function (shouldn't be referenced as ip though).
>
> What's rbx? Is the memory at %rbx + 0x30 valid?
>
> Also, did you by any chance upgrade the binaries while the code was running? is the code running over nfs?
>
> Yehuda
>
>>>
>>> Yehuda
>>>
>>>
>>>>>> libfcgi.so.0.0.0[7ffa06995000+a000] in
>>>>>> libfcgi.so.0.0.0[7ffa06995000+a000]
>>>>>>
>>>>>> Looking at the assembly, it seems crashing at this point -
>>>>>> http://github.com/sknown/fcgi/blob/master/libfcgi/fcgiapp.c#L2035, which
>>>>>> confused me. I tried to see if there is any other reference holding the
>>>>>> FCGX_Request which release the handle without any luck.
>>>>>>
>>>>>> There are also other observations:
>>>>>> 1> Several radosgw daemon across different hosts crashed around the same
>>>>>> time.
>>>>>> 2> Apache's error log has some fcgi error complaining ##idle timeout##
>>>>>> during the time.
>>>>>>
>>>>>> Does anyone experience similar issue?
>>>>>>
>>>>>
>>>>> In the past we've had issues with libfcgi that were related to the number
>>>>> of open fds on the process (> 1024). The issue was a buggy libfcgi that
>>>>> was using select() instead of poll(), so this might be the issue you're
>>>>> noticing.
>>>>>
>>>>> Yehuda
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>> N嫥叉靣笡y氊b瞂千v豝�藓{.n�壏渮榏z鳐妠ay�蕠跈�jf＂穐殝鄗�畐ア�⒎:+v墾妛鑚豰稛�珣赙zZ+凒殠娸"濟!秈
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
|
