Hi,
(2015/06/27 3:33), dwalker@xxxxxxxxxx wrote:> On Fri, Jun 26, 2015 at 10:02:00AM +0900, Hidehiro Kawai wrote:
>> Hi,
>>
>> (2015/06/25 1:31), dwalker@xxxxxxxxxx wrote:
>>> On Wed, Jun 24, 2015 at 03:52:48PM +0900, Masami Hiramatsu wrote:
>>>> Hi,
>>>>
>>>> On 2015/06/23 23:05, dwalker@xxxxxxxxxx wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> There was a commit in kernel/panic.c which altered when the kexec crash kernel is executed,
>>>>>
>>>>> commit f06e5153f4ae2e2f3b0300f0e260e40cb7fefd45
>>>>> Author: Masami Hiramatsu <masami.hiramatsu.pt@xxxxxxxxxxx>
>>>>> Date: Fri Jun 6 14:37:07 2014 -0700
>>>>>
>>>>> kernel/panic.c: add "crash_kexec_post_notifiers" option for kdump after panic_notifers
>>>>>
>>>>>
>>>>> This made it possible for smp_send_stop() to be executed prior to calling the kexec crash
>>>>> kernel.
>>>>>
>>>>> The issue is that smp_send_stop() offlines the cores, and other code depend on the cores being online.
>>>>>
>>>>> In my case on Octeon here's an example,
>>>>>
>>>>> panic()
>>>>> crash_kexec()
>>>>> machine_crash_shutdown()
>>>>> octeon_generic_shutdown()
>>>>>
>>>>> Inside octeon_generic_shutdown() the Octeon watchdog is shutdown for_each_online_cpu(), but since
>>>>> most of the cpu's already got offlined in smp_send_stop() it means the watchdog is still alive on
>>>>> those cores. This results in a reboot during the crash kernel execution.
>>>>
>>>> Ah, I see.
>>>>
>>>>> Another example seem to be in default_machine_crash_shutdown() where crash_kexec_prepare_cpus() depends
>>>>> on an IPI for saving the registers on different cores. However, the cpu's are all offlined with
>>>>> interrupts disabled so they won't be running those IPI's in this case.
>>>>>
>>>>> I'm looking for any advice on how this should be fixed, or if it's already fixed. I'm not going to be
>>>>> submitting a patch so if anyone wants to submit one feel free to do so.
>>>>
>>>> Hmm, IMHO, when the cpu goes to offline in appropriate way(smp_send_stop), it should stop
>>>> watchdog timer on the offlined cpu too.
>>>> Or, you can also register crash handler which stops all watchdogs, but it's a bit tricky.
>>>>
>>>
>>> That doesn't really fix all the issue tho. As I was explaining generic MIPS code depends on the cpu's
>>> effectively being online for crash data collection (with an IPI). This issue may effect other architectures also,
>>> because smp_send_stop() offlines the cpu on other architectures also. I haven't surveyed the other architectures
>>> enough to know what issue could happen from this tho.
>>>
>>> Is it possible to move the smp_send_stop() below the notifiers ? I'm just throwing out ideas.
>>
>> No, that doesn't works. Some notifiers assume that they run in
>> single core mode.
>>
>> Another possible solution is to add notifiers just after
>> machine_crash_shutdown() like this:
>>
>> void panic(const char *fmt, ...)
>> ...
>> - if (!crash_kexec_post_notifiers)
>> - crash_kexec(NULL);
>> + crash_kexec(NULL, buf);
>>
>> and
>>
>> -void crash_kexec(struct pt_regs *regs)
>> +void crash_kexec(struct pt_regs *regs, char *msg)
>> ...
>> if (kexec_crash_image) {
>> struct pt_regs fixed_regs;
>>
>> crash_setup_regs(&fixed_regs, regs);
>> crash_save_vmcoreinfo();
>> machine_crash_shutdown(&fixed_regs);
>> + if (crash_kexec_post_notifiers) {
>> + kmsg_dump(KMSG_DUMP_PANIC);
>> + atomic_notifier_call_chain(&panic_notifier_list, 0, msg);
>> + }
>> machine_kexec(kexec_crash_image);
>>
>> Most of archs stop other cores in machine_crash_shutdown(),
>> so it will work well. Furthermore, it simplifies the special
>> case where crash_kexec() is called without entering panic().
>>
>> However, we need some tweaks for sh and s390 cases. As for sh,
>> it seems not to stop other cores in the crash_kexec() sequence
>> (kdump support is incompleted?). For s390, smp_send_stop() is
>> called in machine_kexec() but not machine_crash_shutdown().
>
> You could add an ifdef into the __setup() to filter out s390 and sh, until we figure out what
> to do there. So the "crash_kexec_post_notifiers" wouldn't be available for those platforms.
I agree on disabling the "crash_kexec_post_notifiers" feature for
s390 and sh at this time. Also, we should make this feature effective
only if CONFIG_CRASH_DUMP=y. Otherwise, it makes no sense.
I'll prepare the bug fix patch. Please wait a moment.
Regards,
--
Hidehiro Kawai
Hitachi, Ltd. Research & Development Group
