Hi Jason,
On Fri, Mar 22, 2024 at 6:31 AM Jason Wang <jasowang@xxxxxxxxxx> wrote:
>
> On Thu, Mar 21, 2024 at 5:44 PM Igor Raits <igor@xxxxxxxxxxxx> wrote:
> >
> > Hello Jason & others,
> >
> > On Wed, Mar 20, 2024 at 10:33 AM Jason Wang <jasowang@xxxxxxxxxx> wrote:
> > >
> > > On Tue, Mar 19, 2024 at 9:15 PM Igor Raits <igor@xxxxxxxxxxxx> wrote:
> > > >
> > > > Hello Stefan,
> > > >
> > > > On Tue, Mar 19, 2024 at 2:12 PM Stefan Hajnoczi <stefanha@xxxxxxxxxx> wrote:
> > > > >
> > > > > On Tue, Mar 19, 2024 at 10:00:08AM +0100, Igor Raits wrote:
> > > > > > Hello,
> > > > > >
> > > > > > We have started to observe kernel crashes on 6.7.y kernels (atm we
> > > > > > have hit the issue 5 times on 6.7.5 and 6.7.10). On 6.6.9 where we
> > > > > > have nodes of cluster it looks stable. Please see stacktrace below. If
> > > > > > you need more information please let me know.
> > > > > >
> > > > > > We do not have a consistent reproducer but when we put some bigger
> > > > > > network load on a VM, the hypervisor's kernel crashes.
> > > > > >
> > > > > > Help is much appreciated! We are happy to test any patches.
> > > > >
> > > > > CCing Michael Tsirkin and Jason Wang for vhost_net.
> > > > >
> > > > > >
> > > > > > [62254.167584] stack segment: 0000 [#1] PREEMPT SMP NOPTI
> > > > > > [62254.173450] CPU: 63 PID: 11939 Comm: vhost-11890 Tainted: G
> > > > > > E 6.7.10-1.gdc.el9.x86_64 #1
> > > > >
> > > > > Are there any patches in this kernel?
> > > >
> > > > Only one, unrelated to this part. Removal of pr_err("EEVDF scheduling
> > > > fail, picking leftmost\n"); line (reported somewhere few months ago
> > > > and it was suggested workaround until proper solution comes).
> > >
> > > Btw, a bisection would help as well.
> >
> > In the end it seems like we don't really have "stable" setup, so
> > bisection looks to be useless but we did find few things meantime:
> >
> > 1. On 6.6.9 it crashes either with unexpected GSO type or usercopy:
> > Kernel memory exposure attempt detected from SLUB object
> > 'skbuff_head_cache'
>
> Do you have a full calltrace for this?
I have shared it in one of the messages in this thread.
https://marc.info/?l=linux-virtualization&m=171085443512001&w=2
> > 2. On 6.7.5, 6.7.10 and 6.8.1 it crashes with RIP:
> > 0010:skb_release_data+0xb8/0x1e0
>
> And for this?
https://marc.info/?l=linux-netdev&m=171083870801761&w=2
> > 3. It does NOT crash on 6.8.1 when VM does not have multi-queue setup
> >
> > Looks like the multi-queue setup (we have 2 interfaces × 3 virtio
> > queues for each) is causing problems as if we set only one queue for
> > each interface the issue is gone.
> > Maybe there is some race condition in __pfx_vhost_task_fn+0x10/0x10 or
> > somewhere around?
>
> I can't tell now, but it seems not because if we have 3 queue pairs we
> will have 3 vhost threads.
>
> > We have noticed that there are 3 of such functions
> > in the stacktrace that gave us hints about what we could try…
>
> Let's try to enable SLUB_DEBUG and KASAN to see if we can get
> something interesting.
We were able to reproduce it even with 1 vhost queue... And now we
have slub_debug + kasan so I hopefully have more useful data for you
now.
I have attached it for better readability.
Thanks!
cmp0220 login: [ 673.018736] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
[ 905.913602] tun: Universal TUN/TAP device driver, 1.6
[ 905.922523] brqf5da63bf-9a: port 1(tap07736b53-0b) entered blocking state
[ 905.929427] brqf5da63bf-9a: port 1(tap07736b53-0b) entered disabled state
[ 905.936288] tap07736b53-0b: entered allmulticast mode
[ 905.941869] tap07736b53-0b: entered promiscuous mode
[ 905.947932] brqf5da63bf-9a: port 1(tap07736b53-0b) entered blocking state
[ 905.954795] brqf5da63bf-9a: port 1(tap07736b53-0b) entered forwarding state
[ 906.012320] brqfd2aebce-2e: port 1(tape61d8e94-b9) entered blocking state
[ 906.019228] brqfd2aebce-2e: port 1(tape61d8e94-b9) entered disabled state
[ 906.026080] tape61d8e94-b9: entered allmulticast mode
[ 906.033505] tape61d8e94-b9: entered promiscuous mode
[ 906.039472] brqfd2aebce-2e: port 1(tape61d8e94-b9) entered blocking state
[ 906.046324] brqfd2aebce-2e: port 1(tape61d8e94-b9) entered forwarding state
[ 906.881154] brqf5da63bf-9a: port 2(bond0.711) entered blocking state
[ 906.887611] brqf5da63bf-9a: port 2(bond0.711) entered disabled state
[ 906.894061] bond0.711: entered allmulticast mode
[ 906.899239] bond0.711: entered promiscuous mode
[ 906.904958] brqf5da63bf-9a: port 2(bond0.711) entered blocking state
[ 906.911367] brqf5da63bf-9a: port 2(bond0.711) entered forwarding state
cmp0220 login:
cmp0220 login:
cmp0220 login: [ 908.571260] brqfd2aebce-2e: port 2(bond0.700) entered blocking state
[ 908.577691] brqfd2aebce-2e: port 2(bond0.700) entered disabled state
[ 908.584186] bond0.700: entered allmulticast mode
[ 908.589575] bond0.700: entered promiscuous mode
[ 908.594579] brqfd2aebce-2e: port 2(bond0.700) entered blocking state
[ 908.601442] brqfd2aebce-2e: port 2(bond0.700) entered forwarding state
[ 944.310935] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered blocking state
[ 944.318255] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered disabled state
[ 944.325566] tapd18e017d-f0: entered allmulticast mode
[ 944.331738] tapd18e017d-f0: entered promiscuous mode
[ 944.338187] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered blocking state
[ 944.345689] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered forwarding state
[ 944.386641] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered blocking state
[ 944.393932] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered disabled state
[ 944.401313] tap01bbfc9f-e4: entered allmulticast mode
[ 944.407819] tap01bbfc9f-e4: entered promiscuous mode
[ 944.414431] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered blocking state
[ 944.421799] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered forwarding state
cmp0220 login:
cmp0220 login:
cmp0220 login:
cmp0220 login: [ 956.794495] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered disabled state
[ 956.805423] tap01bbfc9f-e4 (unregistering): left allmulticast mode
[ 956.812555] tap01bbfc9f-e4 (unregistering): left promiscuous mode
[ 956.819402] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered disabled state
[ 956.861804] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered disabled state
[ 956.869911] tapd18e017d-f0 (unregistering): left allmulticast mode
[ 956.876562] tapd18e017d-f0 (unregistering): left promiscuous mode
[ 956.883126] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered disabled state
[ 1056.270117] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered blocking state
[ 1056.277499] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered disabled state
[ 1056.284753] tapd18e017d-f0: entered allmulticast mode
[ 1056.291061] tapd18e017d-f0: entered promiscuous mode
[ 1056.297223] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered blocking state
[ 1056.304527] brqf5da63bf-9a: port 3(tapd18e017d-f0) entered forwarding state
[ 1056.343876] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered blocking state
[ 1056.351076] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered disabled state
[ 1056.358841] tap01bbfc9f-e4: entered allmulticast mode
[ 1056.364645] tap01bbfc9f-e4: entered promiscuous mode
[ 1056.370601] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered blocking state
[ 1056.377799] brqfd2aebce-2e: port 3(tap01bbfc9f-e4) entered forwarding state
cmp0220 login:
cmp0220 login:
cmp0220 login: [ 1339.821946] systemd-journald[1873]: Data hash table of /run/log/journal/ecd10828def34e4d93adc83211012bf3/system.journal has a fill level at 75.0 (17071 of 22755 items, 8388608 file size, 491 bytes per hash table item), suggesting rotation.
[ 1339.844609] systemd-journald[1873]: /run/log/journal/ecd10828def34e4d93adc83211012bf3/system.journal: Journal header limits reached or header out-of-date, rotating.
cmp0220 login:
cmp0220 login:
cmp0220 login:
cmp0220 login:
cmp0220 login: [ 1367.335154] ==================================================================
[ 1367.342911] BUG: KASAN: use-after-free in _copy_to_iter+0x378/0x10b0
[ 1367.349749] Read of size 3006 at addr ffff88b07d92c102 by task vhost-18439/18512
[ 1367.357635]
[ 1367.359602] CPU: 5 PID: 18512 Comm: vhost-18439 Tainted: G E 6.8.1-1.gdc.el9.kasan.x86_64 #1
[ 1367.369907] Hardware name: Dell Inc. PowerEdge R7525/0H3K7P, BIOS 2.14.1 12/17/2023
[ 1367.378040] Call Trace:
[ 1367.380970] <TASK>
[ 1367.383549] dump_stack_lvl+0x33/0x50
[ 1367.387694] print_address_description.constprop.0+0x2c/0x3e0
[ 1367.393918] ? _copy_to_iter+0x378/0x10b0
[ 1367.398418] print_report+0xb5/0x270
[ 1367.402470] ? kasan_addr_to_slab+0x9/0xa0
[ 1367.407050] kasan_report+0xac/0xe0
[ 1367.411011] ? _copy_to_iter+0x378/0x10b0
[ 1367.415498] kasan_check_range+0x35/0x1b0
[ 1367.419980] _copy_to_iter+0x378/0x10b0
[ 1367.424285] ? kasan_save_stack+0x30/0x40
[ 1367.428769] ? kasan_save_stack+0x20/0x40
[ 1367.433251] ? kasan_save_track+0x10/0x30
[ 1367.437722] ? kasan_save_free_info+0x37/0x60
[ 1367.442533] ? poison_slab_object+0x106/0x190
[ 1367.447339] ? __pfx__copy_to_iter+0x10/0x10
[ 1367.452063] ? tun_recvmsg+0x10b/0x380 [tun]
[ 1367.456790] ? check_heap_object+0x27d/0x4b0
[ 1367.461508] ? ret_from_fork_asm+0x1b/0x30
[ 1367.466059] ? __check_object_size.part.0+0x66/0x1c0
[ 1367.471474] __skb_datagram_iter+0x405/0x8f0
[ 1367.476198] ? __pfx_simple_copy_to_iter+0x10/0x10
[ 1367.481433] skb_copy_datagram_iter+0x38/0x100
[ 1367.486324] tun_put_user.constprop.0+0x440/0xba0 [tun]
[ 1367.491998] ? __pfx_vhost_get_vq_desc+0x10/0x10 [vhost]
[ 1367.497774] ? __pfx_tun_put_user.constprop.0+0x10/0x10 [tun]
[ 1367.503975] tun_do_read+0xba/0x3d0 [tun]
[ 1367.508444] ? __pfx_tun_do_read+0x10/0x10 [tun]
[ 1367.513518] ? __pfx__copy_to_iter+0x10/0x10
[ 1367.518241] ? __pfx_get_rx_bufs+0x10/0x10 [vhost_net]
[ 1367.523836] ? vhost_net_buf_peek+0x448/0x6f0 [vhost_net]
[ 1367.529687] tun_recvmsg+0x10b/0x380 [tun]
[ 1367.534254] handle_rx+0x81d/0x1120 [vhost_net]
[ 1367.539243] ? __pfx_handle_rx+0x10/0x10 [vhost_net]
[ 1367.544669] ? __schedule+0x6cd/0x1760
[ 1367.548873] ? mutex_unlock+0x80/0xd0
[ 1367.552994] ? __pfx_mutex_unlock+0x10/0x10
[ 1367.557615] vhost_worker+0xcb/0x110 [vhost]
[ 1367.562318] vhost_task_fn+0x114/0x250
[ 1367.566484] ? __pfx_vhost_task_fn+0x10/0x10
[ 1367.571147] ? _raw_spin_lock_irq+0x82/0xe0
[ 1367.575713] ? __pfx__raw_spin_lock_irq+0x10/0x10
[ 1367.580797] ? finish_task_switch.isra.0+0x1b3/0x720
[ 1367.586141] ? __pfx_vhost_task_fn+0x10/0x10
[ 1367.590780] ret_from_fork+0x2d/0x70
[ 1367.594716] ? __pfx_vhost_task_fn+0x10/0x10
[ 1367.599338] ret_from_fork_asm+0x1b/0x30
[ 1367.603603] </TASK>
[ 1367.606136]
[ 1367.607970] The buggy address belongs to the physical page:
[ 1367.613888] page:ffffea00c1f64b00 refcount:65535 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x307d92c
[ 1367.624079] head:ffffea00c1f64b00 order:1 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[ 1367.632534] flags: 0x97ffffc0000040(head|node=2|zone=2|lastcpupid=0x1fffff)
[ 1367.639865] page_type: 0xffffffff()
[ 1367.643731] raw: 0097ffffc0000040 0000000000000000 dead000000000122 0000000000000000
[ 1367.651857] raw: 0000000000000000 0000000000000000 0000ffffffffffff 0000000000000000
[ 1367.659979] page dumped because: kasan: bad access detected
[ 1367.665941]
[ 1367.667825] Memory state around the buggy address:
[ 1367.673025] ffff88b07d92c000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 1367.680655] ffff88b07d92c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 1367.688277] >ffff88b07d92c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 1367.695901] ^
[ 1367.699531] ffff88b07d92c180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 1367.707161] ffff88b07d92c200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 1367.714792] ==================================================================
[ 1367.722460] Disabling lock debugging due to kernel taint
[ 1369.686357] usercopy: Kernel memory exposure attempt detected from SLUB object 'skbuff_ext_cache' (offset 128, size 3072)!
[ 1369.698318] ------------[ cut here ]------------
[ 1369.703568] kernel BUG at mm/usercopy.c:102!
[ 1369.708421] invalid opcode: 0000 [#1] PREEMPT SMP KASAN NOPTI
[ 1369.714694] CPU: 89 PID: 20746 Comm: vhost-20705 Tainted: G B E 6.8.1-1.gdc.el9.kasan.x86_64 #1
[ 1369.725117] Hardware name: Dell Inc. PowerEdge R7525/0H3K7P, BIOS 2.14.1 12/17/2023
[ 1369.733284] RIP: 0010:usercopy_abort+0x6a/0x80
[ 1369.738239] Code: 18 bc ab 50 48 c7 c2 40 18 bc ab 57 48 c7 c7 40 19 bc ab 48 0f 44 d6 48 c7 c6 80 18 bc ab 4c 89 d1 49 0f 44 f3 e8 96 48 8a ff <0f> 0b 49 c7 c1 c0 17 bc ab 4c 89 cf 4d 89 c8 eb a9 0f 1f 44 00 00
[ 1369.758000] RSP: 0018:ffff8990103b7830 EFLAGS: 00010246
[ 1369.763746] RAX: 000000000000006e RBX: 0000000000000080 RCX: ffffffffaa2c62ba
[ 1369.771400] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff8990103b7658
[ 1369.779052] RBP: 0000000000000c00 R08: 0000000000000001 R09: ffffed3202076ecb
[ 1369.786697] R10: ffff8990103b765f R11: ffff88801c5c6fb4 R12: ffff88d002e32680
[ 1369.794340] R13: 0000000000000001 R14: 0000000000002336 R15: ffff8990103b7d38
[ 1369.802020] FS: 00007fc7b8c54f80(0000) GS:ffff89dfbfc80000(0000) knlGS:0000000000000000
[ 1369.810626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1369.816895] CR2: 000000c0f2b8a000 CR3: 000001100920a002 CR4: 0000000000770ef0
[ 1369.824551] PKRU: 55555554
[ 1369.827786] Call Trace:
[ 1369.830759] <TASK>
[ 1369.833383] ? die+0x33/0x90
[ 1369.836795] ? do_trap+0x19a/0x250
[ 1369.840718] ? usercopy_abort+0x6a/0x80
[ 1369.845066] ? do_error_trap+0x8a/0xf0
[ 1369.849326] ? usercopy_abort+0x6a/0x80
[ 1369.853676] ? handle_invalid_op+0x2c/0x40
[ 1369.858272] ? usercopy_abort+0x6a/0x80
[ 1369.862658] ? exc_invalid_op+0x2b/0x50
[ 1369.867020] ? asm_exc_invalid_op+0x16/0x20
[ 1369.871711] ? llist_add_batch+0xba/0x130
[ 1369.876253] ? usercopy_abort+0x6a/0x80
[ 1369.880571] __check_heap_object+0xd5/0x110
[ 1369.885238] __check_object_size.part.0+0x66/0x1c0
[ 1369.890501] simple_copy_to_iter+0x26/0x50
[ 1369.895072] __skb_datagram_iter+0xf8/0x8f0
[ 1369.899725] ? __pfx__copy_to_iter+0x10/0x10
[ 1369.904456] ? translate_desc+0xe7/0x3e0 [vhost]
[ 1369.909537] ? __pfx_simple_copy_to_iter+0x10/0x10
[ 1369.914788] skb_copy_datagram_iter+0x38/0x100
[ 1369.919686] tun_put_user.constprop.0+0x440/0xba0 [tun]
[ 1369.925376] ? __pfx_vhost_get_vq_desc+0x10/0x10 [vhost]
[ 1369.931162] ? __pfx_tun_put_user.constprop.0+0x10/0x10 [tun]
[ 1369.937374] tun_do_read+0xba/0x3d0 [tun]
[ 1369.941846] ? __pfx_tun_do_read+0x10/0x10 [tun]
[ 1369.946934] ? __pfx__copy_to_iter+0x10/0x10
[ 1369.951667] ? __pfx_get_rx_bufs+0x10/0x10 [vhost_net]
[ 1369.957260] tun_recvmsg+0x10b/0x380 [tun]
[ 1369.961805] handle_rx+0x81d/0x1120 [vhost_net]
[ 1369.966768] ? __pfx_handle_rx+0x10/0x10 [vhost_net]
[ 1369.972160] ? __schedule+0x6cd/0x1760
[ 1369.976345] ? mutex_unlock+0x80/0xd0
[ 1369.980430] ? __pfx_mutex_unlock+0x10/0x10
[ 1369.985027] vhost_worker+0xcb/0x110 [vhost]
[ 1369.989701] vhost_task_fn+0x114/0x250
[ 1369.993835] ? __pfx_vhost_task_fn+0x10/0x10
[ 1369.998470] ? _raw_spin_lock_irq+0x82/0xe0
[ 1370.003017] ? __pfx__raw_spin_lock_irq+0x10/0x10
[ 1370.008078] ? finish_task_switch.isra.0+0x1b3/0x720
[ 1370.013403] ? __pfx_vhost_task_fn+0x10/0x10
[ 1370.018028] ret_from_fork+0x2d/0x70
[ 1370.021960] ? __pfx_vhost_task_fn+0x10/0x10
[ 1370.026584] ret_from_fork_asm+0x1b/0x30
[ 1370.030862] </TASK>
[ 1370.033382] Modules linked in: nf_conntrack_netlink(E) ebt_arp(E) nft_meta_bridge(E) xt_CT(E) xt_mac(E) xt_set(E) xt_conntrack(E) xt_comment(E) xt_physdev(E) nft_compat(E) ip_set_hash_net(E) ip_set(E) vhost_net(E) vhost(E) vhost_iotlb(E) tap(E) tun(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) netconsole(E) ib_core(E) scsi_transport_iscsi(E) nf_tables(E) nfnetlink(E) target_core_mod(E) macvlan(E) 8021q(E) garp(E) mrp(E) bonding(E) tls(E) binfmt_misc(E) dell_rbu(E) sunrpc(E) vfat(E) fat(E) dm_service_time(E) dm_multipath(E) ipmi_ssif(E) btrfs(E) intel_rapl_msr(E) intel_rapl_common(E) xor(E) amd64_edac(E) zstd_compress(E) dell_smbios(E) edac_mce_amd(E) kvm_amd(E) raid6_pq(E) dell_wmi_descriptor(E) wmi_bmof(E) dcdbas(E) kvm(E) irqbypass(E) mgag200(E) i2c_algo_bit(E) rapl(E) acpi_cpufreq(E) ptdma(E) acpi_ipmi(E) i2c_piix4(E) k10temp(E) wmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_power_meter(E) fuse(E) zram(E) ext4(E) mbcache(E) jbd2(E) dm_crypt(E) sd_mod(E)
[ 1370.033475] t10_pi(E) sg(E) crct10dif_pclmul(E) crc32_pclmul(E) polyval_clmulni(E) ahci(E) polyval_generic(E) libahci(E) ghash_clmulni_intel(E) ice(E) sha512_ssse3(E) libata(E) ccp(E) gnss(E) megaraid_sas(E) sp5100_tco(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) nf_conntrack(E) libcrc32c(E) crc32c_intel(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) br_netfilter(E) bridge(E) stp(E) llc(E)
[ 1370.127042] Unloaded tainted modules: fjes(E):2 padlock_aes(E):3
[ 1370.170306] ---[ end trace 0000000000000000 ]---
[ 1370.231095] pstore: backend (erst) writing error (-22)
[ 1370.236814] RIP: 0010:usercopy_abort+0x6a/0x80
[ 1370.241826] Code: 18 bc ab 50 48 c7 c2 40 18 bc ab 57 48 c7 c7 40 19 bc ab 48 0f 44 d6 48 c7 c6 80 18 bc ab 4c 89 d1 49 0f 44 f3 e8 96 48 8a ff <0f> 0b 49 c7 c1 c0 17 bc ab 4c 89 cf 4d 89 c8 eb a9 0f 1f 44 00 00
[ 1370.261760] RSP: 0018:ffff8990103b7830 EFLAGS: 00010246
[ 1370.267600] RAX: 000000000000006e RBX: 0000000000000080 RCX: ffffffffaa2c62ba
[ 1370.275353] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff8990103b7658
[ 1370.283101] RBP: 0000000000000c00 R08: 0000000000000001 R09: ffffed3202076ecb
[ 1370.290847] R10: ffff8990103b765f R11: ffff88801c5c6fb4 R12: ffff88d002e32680
[ 1370.298596] R13: 0000000000000001 R14: 0000000000002336 R15: ffff8990103b7d38
[ 1370.306354] FS: 00007fc7b8c54f80(0000) GS:ffff89dfbfc80000(0000) knlGS:0000000000000000
[ 1370.315067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1370.321439] CR2: 000000c0f2b8a000 CR3: 000001100920a002 CR4: 0000000000770ef0
[ 1370.329204] PKRU: 55555554
[ 1370.332543] Kernel panic - not syncing: Fatal exception
[ 1370.476941] Kernel Offset: 0x28200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
