On Mon, Jun 29, 2015 at 06:53:29PM +0900, Hidehiro Kawai wrote: > Hi, > > (2015/06/27 3:33), dwalker@xxxxxxxxxx wrote:> On Fri, Jun 26, 2015 at 10:02:00AM +0900, Hidehiro Kawai wrote: > >> Hi, > >> > >> (2015/06/25 1:31), dwalker@xxxxxxxxxx wrote: > >>> On Wed, Jun 24, 2015 at 03:52:48PM +0900, Masami Hiramatsu wrote: > >>>> Hi, > >>>> > >>>> On 2015/06/23 23:05, dwalker@xxxxxxxxxx wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> There was a commit in kernel/panic.c which altered when the kexec crash kernel is executed, > >>>>> > >>>>> commit f06e5153f4ae2e2f3b0300f0e260e40cb7fefd45 > >>>>> Author: Masami Hiramatsu <masami.hiramatsu.pt@xxxxxxxxxxx> > >>>>> Date: Fri Jun 6 14:37:07 2014 -0700 > >>>>> > >>>>> kernel/panic.c: add "crash_kexec_post_notifiers" option for kdump after panic_notifers > >>>>> > >>>>> > >>>>> This made it possible for smp_send_stop() to be executed prior to calling the kexec crash > >>>>> kernel. > >>>>> > >>>>> The issue is that smp_send_stop() offlines the cores, and other code depend on the cores being online. > >>>>> > >>>>> In my case on Octeon here's an example, > >>>>> > >>>>> panic() > >>>>> crash_kexec() > >>>>> machine_crash_shutdown() > >>>>> octeon_generic_shutdown() > >>>>> > >>>>> Inside octeon_generic_shutdown() the Octeon watchdog is shutdown for_each_online_cpu(), but since > >>>>> most of the cpu's already got offlined in smp_send_stop() it means the watchdog is still alive on > >>>>> those cores. This results in a reboot during the crash kernel execution. > >>>> > >>>> Ah, I see. > >>>> > >>>>> Another example seem to be in default_machine_crash_shutdown() where crash_kexec_prepare_cpus() depends > >>>>> on an IPI for saving the registers on different cores. However, the cpu's are all offlined with > >>>>> interrupts disabled so they won't be running those IPI's in this case. > >>>>> > >>>>> I'm looking for any advice on how this should be fixed, or if it's already fixed. I'm not going to be > >>>>> submitting a patch so if anyone wants to submit one feel free to do so. > >>>> > >>>> Hmm, IMHO, when the cpu goes to offline in appropriate way(smp_send_stop), it should stop > >>>> watchdog timer on the offlined cpu too. > >>>> Or, you can also register crash handler which stops all watchdogs, but it's a bit tricky. > >>>> > >>> > >>> That doesn't really fix all the issue tho. As I was explaining generic MIPS code depends on the cpu's > >>> effectively being online for crash data collection (with an IPI). This issue may effect other architectures also, > >>> because smp_send_stop() offlines the cpu on other architectures also. I haven't surveyed the other architectures > >>> enough to know what issue could happen from this tho. > >>> > >>> Is it possible to move the smp_send_stop() below the notifiers ? I'm just throwing out ideas. > >> > >> No, that doesn't works. Some notifiers assume that they run in > >> single core mode. > >> > >> Another possible solution is to add notifiers just after > >> machine_crash_shutdown() like this: > >> > >> void panic(const char *fmt, ...) > >> ... > >> - if (!crash_kexec_post_notifiers) > >> - crash_kexec(NULL); > >> + crash_kexec(NULL, buf); > >> > >> and > >> > >> -void crash_kexec(struct pt_regs *regs) > >> +void crash_kexec(struct pt_regs *regs, char *msg) > >> ... > >> if (kexec_crash_image) { > >> struct pt_regs fixed_regs; > >> > >> crash_setup_regs(&fixed_regs, regs); > >> crash_save_vmcoreinfo(); > >> machine_crash_shutdown(&fixed_regs); > >> + if (crash_kexec_post_notifiers) { > >> + kmsg_dump(KMSG_DUMP_PANIC); > >> + atomic_notifier_call_chain(&panic_notifier_list, 0, msg); > >> + } > >> machine_kexec(kexec_crash_image); > >> > >> Most of archs stop other cores in machine_crash_shutdown(), > >> so it will work well. Furthermore, it simplifies the special > >> case where crash_kexec() is called without entering panic(). > >> > >> However, we need some tweaks for sh and s390 cases. As for sh, > >> it seems not to stop other cores in the crash_kexec() sequence > >> (kdump support is incompleted?). For s390, smp_send_stop() is > >> called in machine_kexec() but not machine_crash_shutdown(). > > > > You could add an ifdef into the __setup() to filter out s390 and sh, until we figure out what > > to do there. So the "crash_kexec_post_notifiers" wouldn't be available for those platforms. > > I agree on disabling the "crash_kexec_post_notifiers" feature for > s390 and sh at this time. Also, we should make this feature effective > only if CONFIG_CRASH_DUMP=y. Otherwise, it makes no sense. > > I'll prepare the bug fix patch. Please wait a moment. Why to add a patch for that. Just inform the user that don't need crash_kexec_post_notifiers on s390 and sh? Thanks Vivek