> -----Original Message-----
> From: Michael Kelley <mhklinux@xxxxxxxxxxx>
> Sent: Thursday, January 30, 2025 3:20 PM
> To: Thomas Tai <thomas.tai@xxxxxxxxxx>; mhkelley58@xxxxxxxxx;
> haiyangz@xxxxxxxxxxxxx; wei.liu@xxxxxxxxxx; decui@xxxxxxxxxxxxx;
> drawat.floss@xxxxxxxxx; javierm@xxxxxxxxxx; Helge Deller
> <deller@xxxxxx>; daniel@xxxxxxxx; airlied@xxxxxxxxx;
> tzimmermann@xxxxxxx
> Cc: dri-devel@xxxxxxxxxxxxxxxxxxxxx; linux-fbdev@xxxxxxxxxxxxxxx; linux-
> kernel@xxxxxxxxxxxxxxx; linux-hyperv@xxxxxxxxxxxxxxx
> Subject: RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
> executing kexec
>
> From: Thomas Tai <thomas.tai@xxxxxxxxxx> Sent: Thursday, January 30,
> 2025 10:50 AM
> >
> > Sorry for the typo in the subject title. It should have been 'hyperv_fb soft
> lockup on
> > Azure Gen2 VM when taking kdump or executing kexec'
> >
> > Thomas
> >
> > >
> > > Hi Michael,
> > >
> > > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > > trying to induce a kernel panic with sysrq commands. The VM would
> hang
> > > with soft lockup. A similar issue happens when executing kexec on the
> VM.
> > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with
> bios
> > > boot are fine.
> > >
> > > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > However, reverting the commit would cause the frame buffer not to
> work
> > > on the Gen2 VM.
> > >
> > > Do you have any hints on what caused this issue?
> > >
> > > To reproduce the issue with kdump:
> > > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > > - echo 1 > /proc/sys/kernel/sysrq
> > > - echo c > /proc/sysrq-trigger
> > >
> > > To reproduce the issue with executing kexec:
> > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > > line="$( cat /proc/cmdline )"
> > > - sudo kexec -e
> > >
> > > Thank you,
> > > Thomas
>
> I will take a look, but it might be early next week before I can do so.
>
Thank you, Michael for your help!
> It looks like your soft lockup log below is from the kdump kernel (or the
> newly
> kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full
> log.
Yes, the soft lockup log below is from the kdump kernel.
> Do you have the full serial console log that you could email to me? Seeing
> everything might be helpful. Of course, I'll try to repro the problem myself
> as
> well.
I have attached the complete bootup and kdump kernel log.
File: bootup_and_kdump.log
Line 1 ... 984 (bootup log)
Line 990 (kdump kernel booting up)
Line 1351 (soft lockup)
Thank you,
Thomas
>
> Thanks,
>
> Michael
>
> > >
> > > --- soft lockup log---
> > > [ 1.690032] efifb: probing for efifb
> > > [ 1.693989] efifb: framebuffer at 0x40000000, using 3072k, total 3072k
> > > [ 1.700996] efifb: mode is 1024x768x32, linelength=4096, pages=1
> > > [ 1.706999] efifb: scrolling: redraw
> > > [ 1.710981] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> > > [ 1.716806] Console: switching to colour frame buffer device 128x48
> > > [ 29.176649] watchdog: BUG: soft lockup - CPU#0 stuck for 26s!
> > > [swapper/0:1]
> > > [ 29.176655] Modules linked in:
> > > [ 29.176658] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.13.0+
> #1
> > > [ 29.176661] Hardware name: Microsoft Corporation Virtual
> > > Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
> > > [ 29.176663] RIP: 0010:fast_imageblit.isra.0+0x2de/0x460
> > > [ 29.176669] Code: 07 49 21 f1 41 83 e1 01 46 8b 4c 8c 40 44 89 09 41 89
> d1
> > > 4c 8d 79 08 41 c0 e9 06 49 21 f1 41 83 e1 03 46 8b 4c 8c 40 45 89 0e <41>
> 89
> > > d1 4c 8d 71 0c 41 c0 e9 05 49 21 f1 41 83 e1 07 46 8b 4c 8c
> > > [ 29.176671] RSP: 0018:ffffc900000437a8 EFLAGS: 00010246
> > > [ 29.176673] RAX: 0000000000000400 RBX: 0000000000000005 RCX:
> > > ffffc900006bb140
> > > [ 29.176674] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> > > ffff888012146600
> > > [ 29.176675] RBP: 0000000000000080 R08: ffffc900006bb000 R09:
> > > 0000000000000000
> > > [ 29.176676] R10: 0000000000000001 R11: ffff888012146580 R12:
> > > 0000000000001000
> > > [ 29.176677] R13: ffff88801214658a R14: ffffc900006bb144 R15:
> > > ffffc900006bb148
> > > [ 29.176678] FS: 0000000000000000(0000) GS:ffff888026a00000(0000)
> > > knlGS:0000000000000000
> > > [ 29.176680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 29.176681] CR2: 00007f8bc160c238 CR3: 000000002ac38002 CR4:
> > > 00000000003706b0
> > > [ 29.176683] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 29.176684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > > 0000000000000400
> > > [ 29.176685] Call Trace:
> > > [ 29.176687] <IRQ>
> > > [ 29.176690] ? watchdog_timer_fn+0x220/0x2a0
> > > [ 29.176695] ? __pfx_watchdog_timer_fn+0x10/0x10
> > > [ 29.176697] ? __hrtimer_run_queues+0x112/0x2b0
> > > [ 29.176702] ? hrtimer_interrupt+0x108/0x270
> > > [ 29.176704] ? sched_clock_cpu+0x60/0x1a0
> > > [ 29.176708] ? __sysvec_hyperv_stimer0+0x32/0x60
> > > [ 29.176712] ? sysvec_hyperv_stimer0+0x70/0x90
> > > [ 29.176715] </IRQ>
> > > [ 29.176716] <TASK>
> > > [ 29.176716] ? asm_sysvec_hyperv_stimer0+0x1a/0x20
> > > [ 29.176722] ? fast_imageblit.isra.0+0x2de/0x460
> > > [ 29.176724] cfb_imageblit+0x433/0x470
> > > [ 29.176726] bit_putcs+0x291/0x570
> > > [ 29.176731] ? __pfx_bit_putcs+0x10/0x10
> > > [ 29.176733] fbcon_putcs+0x139/0x1a0
> > > [ 29.176735] do_update_region+0xf1/0x110
> > > [ 29.176740] redraw_screen+0x22f/0x290
> > > [ 29.176743] do_bind_con_driver.isra.0+0x2ab/0x3d0
> > > [ 29.176745] do_take_over_console+0x3a/0x50
> > > [ 29.176747] do_fbcon_takeover+0x5c/0xe0
> > > [ 29.176749] fbcon_fb_registered+0x4f/0x70
> > > [ 29.176751] do_register_framebuffer+0x1bc/0x2a0
> > > [ 29.176755] devm_register_framebuffer+0x28/0x90
> > > [ 29.176757] efifb_probe+0x544/0x720
> > > [ 29.176760] platform_probe+0x43/0xb0
> > > [ 29.176763] really_probe+0xd9/0x390
> > > [ 29.176767] ? __pfx___device_attach_driver+0x10/0x10
> > > [ 29.176769] __driver_probe_device+0x78/0x160
> > > [ 29.176771] driver_probe_device+0x1e/0xa0
> > > [ 29.176773] __device_attach_driver+0x99/0x130
> > > [ 29.176775] bus_for_each_drv+0x98/0xf0
> > > [ 29.176777] __device_attach+0xbc/0x1f0
> > > [ 29.176779] bus_probe_device+0x8d/0xb0
> > > [ 29.176781] device_add+0x4f1/0x6e0
> > > [ 29.176785] platform_device_add+0xfa/0x260
> > > [ 29.176787] sysfb_init+0x109/0x120
> > > [ 29.176791] ? __pfx_sysfb_init+0x10/0x10
> > > [ 29.176793] do_one_initcall+0x5b/0x330
> > > [ 29.176796] do_initcalls+0xac/0x130
> > > [ 29.176800] kernel_init_freeable+0x134/0x1e0
> > > [ 29.176802] ? __pfx_kernel_init+0x10/0x10
> > > [ 29.176806] kernel_init+0x1a/0x1d0
> > > [ 29.176808] ret_from_fork+0x34/0x50
> > > [ 29.176813] ? __pfx_kernel_init+0x10/0x10
> > > [ 29.176815] ret_from_fork_asm+0x1a/0x30
> > > [ 29.176819] </TASK>
> > > [ 432.306986] fb0: EFI VGA frame buffer device
Attachment:
bootup_and_kdump.log
Description: bootup_and_kdump.log
