Hello,
My virtual machine crashed with the message
"0010:virtnet_rq_alloc+0x8f/0x1b0 [virtio_net]". See the full log
below.
I did two changes:
* Updated my VM packages (kernel from 6.9.5 to 6.10.7) on the VM
* enabled "packed virtqueues" by libvirt on host
and it happens after a few hours of uptime.
Any hint how to prevent it or fix this issue?
[52890.265362] BUG: unable to handle page fault for address: ffff9b94c480000c
[52890.266264] #PF: supervisor write access in kernel mode
[52890.266814] #PF: error_code(0x000b) - reserved bit violation
[52890.267299] PGD 4c3c01067 P4D 4c3c01067 PUD 103be6063 PMD 7849cc063
PTE 7a1dd28f4e77cee7
[52890.267926] Oops: Oops: 000b [#1] PREEMPT SMP NOPTI
[52890.268372] CPU: 14 PID: 0 Comm: swapper/14 Tainted: G E
6.10.7-1.gdc.el9.x86_64 #1
[52890.269007] Hardware name: RDO OpenStack Compute/RHEL, BIOS
edk2-20240524-1.el9 05/24/2024
[52890.269853] RIP: 0010:virtnet_rq_alloc+0x8f/0x1b0 [virtio_net]
[52890.270349] Code: e7 e8 85 ef ff ff 49 c7 84 24 80 05 00 00 00 00
00 00 41 0f b7 84 24 ac 02 00 00 48 8d 73 10 45 31 c0 b9 02 00 00 00
8d 50 f0 <66> 89 53 0c 49 8b 3c 24 0f b7 d2 e8 f1 04 9d cb 49 8b 3c 24
48 89
[52890.272173] RSP: 0018:ffffb5dac64f0d60 EFLAGS: 00010246
[52890.272637] RAX: 0000000000001000 RBX: ffff9b94c4800000 RCX: 0000000000000002
[52890.273209] RDX: 0000000000000ff0 RSI: ffff9b94c4800010 RDI: ffffe06380000000
[52890.273772] RBP: 0000000000001000 R08: 0000000000000000 R09: ffffe0639e120008
[52890.274341] R10: 0000000000000000 R11: 00000000000096d0 R12: ffff9b8ebe621800
[52890.274898] R13: ffff9b8ebe621800 R14: 0000000000001000 R15: 0000000000000000
[52890.275476] FS: 0000000000000000(0000) GS:ffff9b9ba3d00000(0000)
knlGS:0000000000000000
[52890.276087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[52890.276592] CR2: ffff9b94c480000c CR3: 00000004c3b1c006 CR4: 0000000000770ef0
[52890.277167] PKRU: 55555554
[52890.277524] Call Trace:
[52890.277869] <IRQ>
[52890.278198] ? __die+0x20/0x70
[52890.278580] ? page_fault_oops+0x75/0x170
[52890.279009] ? exc_page_fault+0xbe/0x160
[52890.279441] ? asm_exc_page_fault+0x22/0x30
[52890.279881] ? virtnet_rq_alloc+0x8f/0x1b0 [virtio_net]
[52890.280377] try_fill_recv+0x22c/0x440 [virtio_net]
[52890.280848] virtnet_receive+0x1ce/0x230 [virtio_net]
[52890.281334] virtnet_poll+0x179/0x3a0 [virtio_net]
[52890.281804] __napi_poll+0x29/0x1b0
[52890.282222] net_rx_action+0x2b5/0x390
[52890.282641] ? _raw_spin_unlock_irqrestore+0xa/0x30
[52890.283118] handle_softirqs+0xd3/0x2b0
[52890.283550] __irq_exit_rcu+0x9b/0xc0
[52890.283970] common_interrupt+0x7f/0xa0
[52890.284409] </IRQ>
[52890.284749] <TASK>
[52890.285084] asm_common_interrupt+0x22/0x40
[52890.285526] RIP: 0010:default_idle+0xb/0x20
[52890.285953] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 90
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 eb 07 0f 00 2d c3 92 30
00 fb f4 <fa> c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
00 90
[52890.287332] RSP: 0018:ffffb5dac436fec0 EFLAGS: 00000206
[52890.287814] RAX: 000000000000000e RBX: ffff9b94c320ce00 RCX: 0000000103227730
[52890.288395] RDX: 000000000000000e RSI: 0000000000000082 RDI: 000000002f3198a4
[52890.288962] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
[52890.289533] R10: 00000000000002cd R11: 0000000000000000 R12: 0000000000000000
[52890.290092] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[52890.290657] default_idle_call+0x2c/0xf0
[52890.291053] cpuidle_idle_call+0x109/0x120
[52890.291464] do_idle+0x76/0xb0
[52890.291813] cpu_startup_entry+0x25/0x30
[52890.292269] start_secondary+0x113/0x130
[52890.292812] common_startup_64+0x13e/0x141
[52890.293279] </TASK>
[52890.293615] Modules linked in: mptcp_diag(E) xsk_diag(E)
raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E)
udp_diag(E) inet_diag(E) nf_conntrack(E) nf_defrag_ipv6(E)
nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) vfat(E)
fat(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E)
i2c_i801(E) virtio_net(E) net_failover(E) failover(E) virtio_gpu(E)
i2c_smbus(E) dimlib(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E)
ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E)
crct10dif_pclmul(E) libahci(E) crc32_pclmul(E) polyval_clmulni(E)
polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E)
virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E)
raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E)
dm_region_hash(E) dm_log(E) dm_mod(E)
[52890.293657] Unloaded tainted modules: edac_mce_amd(E):1
amd_atl(E):2 padlock_aes(E):3
[52890.299716] CR2: ffff9b94c480000c
[52890.300092] ---[ end trace 0000000000000000 ]---
[52890.300101] Oops: general protection fault, probably for
non-canonical address 0x86304c8ed4b709b3: 0000 [#2] PREEMPT SMP NOPTI
[52890.300336] RIP: 0010:virtnet_rq_alloc+0x8f/0x1b0 [virtio_net]
[52890.301177] CPU: 10 PID: 51 Comm: ksoftirqd/10 Tainted: G D
E 6.10.7-1.gdc.el9.x86_64 #1
[52890.301472] Code: e7 e8 85 ef ff ff 49 c7 84 24 80 05 00 00 00 00
00 00 41 0f b7 84 24 ac 02 00 00 48 8d 73 10 45 31 c0 b9 02 00 00 00
8d 50 f0 <66> 89 53 0c 49 8b 3c 24 0f b7 d2 e8 f1 04 9d cb 49 8b 3c 24
48 89
[52890.302137] Hardware name: RDO OpenStack Compute/RHEL, BIOS
edk2-20240524-1.el9 05/24/2024
[52890.303074] RSP: 0018:ffffb5dac64f0d60 EFLAGS: 00010246
[52890.303689] RIP: 0010:put_cpu_partial+0x15/0x70
[52890.303895]
[52890.304291] Code: 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90
90 90 90 90 90 0f 1f 44 00 00 9c 59 fa 48 8b 07 65 4c 8b 40 18 4d 85
c0 74 54 <41> 8b 40 18 85 d2 75 22 83 c0 01 89 46 18 4c 89 46 10 48 8b
07 65
[52890.304556] RAX: 0000000000001000 RBX: ffff9b94c4800000 RCX: 0000000000000002
[52890.304680] RSP: 0018:ffffb5dac6473d48 EFLAGS: 00010082
[52890.305597] RDX: 0000000000000ff0 RSI: ffff9b94c4800010 RDI: ffffe06380000000
[52890.306115]
[52890.306384] RBP: 0000000000001000 R08: 0000000000000000 R09: ffffe0639e120008
[52890.306910] RAX: 00003a3f1c003ff0 RBX: ffff9b99322a16d0 RCX: 0000000000000246
[52890.307010] R10: 0000000000000000 R11: 00000000000096d0 R12: ffff9b8ebe621800
[52890.307574] RDX: 0000000000000001 RSI: ffffe063afc8a800 RDI: ffff9b94cc48fc00
[52890.307927] R13: ffff9b8ebe621800 R14: 0000000000001000 R15: 0000000000000000
[52890.308450] RBP: ffffb5dac6473da0 R08: 86304c8ed4b7099b R09: 00000000001c001b
[52890.309006] FS: 0000000000000000(0000) GS:ffff9b9ba3d00000(0000)
knlGS:0000000000000000
[52890.309543] R10: 0000000000040000 R11: 0000000000000001 R12: ffffe063afc8a840
[52890.310127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[52890.310722] R13: ffffe063afc8a800 R14: ffff9b94cc48fc00 R15: ffff9b9ba3b36100
[52890.311314] CR2: ffff9b94c480000c CR3: 00000004c3b1c006 CR4: 0000000000770ef0
[52890.311735] FS: 0000000000000000(0000) GS:ffff9b9ba3b00000(0000)
knlGS:0000000000000000
[52890.312333] PKRU: 55555554
[52890.312850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[52890.313298] Kernel panic - not syncing: Fatal exception in interrupt
[52891.350776] Shutting down cpus with NMI
[52891.358926] Kernel Offset: 0xae00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[52891.359713] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
Best,
Jaroslav Pulchart
