Hi Gal, On Mon, Mar 25, 2024 at 4:06 PM Gal Pressman <gal.pressman@xxxxxxxxx> wrote: > > On 25/03/2024 4:10, Tao Liu wrote: > > Hi, > > > > Recently I experienced a kernel panic which is related to efa module > > when testing kexec -l && kexec -e to switch to a new kernel on AWS > > i4g.16xlarge instance. > > > > Here is the dmesg log: > > > > [ 6.379918] systemd[1]: Mounting FUSE Control File System... > > [ 6.381984] systemd[1]: Mounting Kernel Configuration File System... > > [ 6.383918] systemd[1]: Starting Apply Kernel Variables... > > [ 6.385430] systemd[1]: Started Journal Service. > > [ 6.394221] ACPI: bus type drm_connector registered > > [ 6.421408] systemd-journald[1263]: Received client request to > > flush runtime journal. > > [ 7.262543] efa 0000:00:1b.0: enabling device (0010 -> 0012) > > [ 7.432420] efa 0000:00:1b.0: Setup irq:191 name:efa-mgmnt@pci:0000:00:1b.0 > > [ 7.435581] efa 0000:00:1b.0 efa_0: IB device registered > > [ 7.885564] random: crng init done > > [ 8.139857] XFS (nvme0n1p2): Mounting V5 Filesystem > > d7003ecc-db6f-4bfb-bf92-60376b6a6563 > > [ 8.265233] XFS (nvme0n1p2): Ending clean mount > > [ 10.555612] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > > > > Red Hat Enterprise Linux 9.4 Beta (Plow) > > Kernel 5.14.0-425.el9.aarch64 on an aarch64 > > > > ip-10-0-27-226 login: [ 29.940381] kexec_core: Starting new kernel > > [ 30.079279] psci: CPU1 killed (polled 0 ms) > > [ 30.119222] psci: CPU2 killed (polled 0 ms) > > [ 30.199293] psci: CPU3 killed (polled 0 ms) > > [ 30.309214] psci: CPU4 killed (polled 0 ms) > > [ 30.379221] psci: CPU5 killed (polled 0 ms) > > [ 30.419210] psci: CPU6 killed (polled 0 ms) > > [ 30.489207] IRQ 191: no longer affine to CPU7 > > [ 30.489667] psci: CPU7 killed (polled 0 ms) > > ..snip... > > [ 33.849123] psci: CPU63 killed (polled 0 ms) > > [ 33.849943] Bye! > > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1] > > [ 0.000000] Linux version 5.14.0-417.el9.aarch64 > > (mockbuild@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) (gcc (GCC) 11.4.1 > > 20231218 (Red Hat 11.4.1-3), GNU ld version 2.35.2-42.el9) #1 SMP > > PREEMPT_DYNAMIC Thu Feb 1 21:23:03 EST 2024 > > ...snip... > > [ 1.012692] Freeing unused kernel memory: 6016K > > [ 2.370947] Checked W+X mappings: passed, no W+X pages found > > [ 2.370980] Run /init as init process > > [ 2.370982] with arguments: > > [ 2.370983] /init > > [ 2.370984] with environment: > > [ 2.370984] HOME=/ > > [ 2.370985] TERM=linux > > [ 2.373257] Kernel panic - not syncing: Attempted to kill init! > > exitcode=0x0000000b > > [ 2.373259] CPU: 1 PID: 1 Comm: init Not tainted 5.14.0-417.el9.aarch64 #1 > > [ 2.382240] Hardware name: Amazon EC2 i4g.16xlarge/, BIOS 1.0 11/1/2018 > > [ 2.383814] Call trace: > > [ 2.384410] dump_backtrace+0xa8/0x120 > > [ 2.385318] show_stack+0x1c/0x30 > > [ 2.386124] dump_stack_lvl+0x74/0x8c > > [ 2.387011] dump_stack+0x14/0x24 > > [ 2.387810] panic+0x158/0x368 > > [ 2.388553] do_exit+0x3a8/0x3b0 > > [ 2.389333] do_group_exit+0x38/0xa4 > > [ 2.390195] get_signal+0x7a4/0x810 > > [ 2.391044] do_signal+0x1bc/0x260 > > [ 2.391870] do_notify_resume+0x108/0x210 > > [ 2.392839] el0_da+0x154/0x160 > > [ 2.393603] el0t_64_sync_handler+0xdc/0x150 > > [ 2.394628] el0t_64_sync+0x17c/0x180 > > [ 2.395513] SMP: stopping secondary CPUs > > [ 2.396483] Kernel Offset: 0x586f04e00000 from 0xffff800008000000 > > [ 2.397934] PHYS_OFFSET: 0x40000000 > > [ 2.398774] CPU features: 0x0,00000101,70020143,10417a0b > > [ 2.400042] Memory Limit: none > > [ 2.400783] ---[ end Kernel panic - not syncing: Attempted to kill > > init! exitcode=0x0000000b ]--- > > > > In the dmesg log, I found "[ 30.489207] IRQ 191: no longer affine to > > CPU7" is suspicious, which is related to efa module. After blacklist > > efa module from automatic loading when bootup, the kernel panic issue > > doesn't appear again. > > > > It looks to me it is due to the efa being not properly shutdown during > > kexec, so the ongoing DMA/interrupts etc overwrite the memory range. > > > > Though the issue is reproduced on rhel's kernel, the upstream kernel > > [1] doesn't have the .shutdown method implemented either. Since I'm > > not very familiar with the efa driver, could you please implement the > > .shutdown method in drivers/infiniband/hw/efa/efa_main.c? Thanks in > > advance! > > Did you try to reproduce it on upstream kernel? > Thanks for your comments! No I haven't, I will give it a try. > > > > [1]: https://github.com/torvalds/linux/blob/master/drivers/infiniband/hw/efa/efa_main.c#L674 > > > > Thanks, > > Tao Liu > > > > Try assigning efa_remove as the shutdown callback: > .shutdown = efa_remove, > > Does it fix it? Thanks, I will also try the code, and I will post the testing results. Thanks, Tao Liu >
