On 2024-05-08 a.m.7:18, Catherine Redfield wrote:
*External email: Use caution opening links or attachments*
On a VM with the GCP kernel (where we first identified the problem), I see:
1. The full kernel log from `journalctl --system > kernlog` attached.
The specific suspend section is here:
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
systemd[1]: Reached target sleep.target - Sleep.
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
systemd[1]: Starting systemd-suspend.service - System Suspend...
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
systemd-sleep[1413]: Performing sleep operation 'suspend'...
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: PM: suspend entry (deep)
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: Filesystems sync: 0.008 seconds
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: Freezing user space processes
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: Freezing user space processes completed (elapsed 0.001 seconds)
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: OOM killer disabled.
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: Freezing remaining freezable tasks
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: Freezing remaining freezable tasks completed (elapsed 0.000 seconds)
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: printk: Suspending console(s) (use no_console_suspend to debug)
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: port 00:03:0.0: PM: dpm_run_callback():
pm_runtime_force_suspend+0x0/0x130 returns -16
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: port 00:03:0.0: PM: failed to suspend: error -16
Thanks Joesph and Catherine's help.
Hi,
I have alreay synced up with Cananical guys offline about this issue.
I can run "suspend/resume" sucessfully on my local server and VM.
And "PM: failed to suspend: error -16" looks like not cause by my
previous virtio patch ( fd27ef6b44be ("virtio-pci: Introduce admin
virtqueue")) which only modified "virtio_device_freeze" about "suspend"
action.
So I have provide the my steps and debug patch to Joesph and Catherine.
I will also sync up the information here, as follow:
I have read the qemu code and find a way to trigger "suspend/resume" on
my setup, and add some debug message in the latest kerenel
My setps are:
1. QEMU cmdline add following
....
-global PIIX4_PM.disable_s3=0 \
-global PIIX4_PM.disable_s4=1 \
....
-netdev type=tap,ifname=tap0,id=hostnet0,script=no,downscript=no \
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=$SSH_MAC,bus=pci.0,addr=0x3 \
......
2. In the VM, run "systemctl suspend" to PM suspend the VM into memory
3. In qemu hmp shell, run "system_wakeup" to resume the VM again
My VM configuration:
NIC: 1 virtio nic emulated by QEMU
OS: Ubuntu 22.04.4 LTS
kernel: latest kernel, 6.9-rc7: ee5b455b0ada (kernel2/net-next-virito,
kernel2/master, master) Merge tag 'slab-for-6.9-rc7-fixes' of
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab)
I add some debug message on the latest kernel, and do above steps to
trigger "suspen/resume". Everything of VM is OK, VM could suspend/resume
successfully.
Follwing is the kernel log:
----------------------------------------------------------------------------
........
May 6 15:59:52 feliu-vm kernel: [ 43.446737] PM: suspend entry (deep)
May 6 16:00:04 feliu-vm kernel: [ 43.467640] Filesystems sync: 0.020
seconds
May 6 16:00:04 feliu-vm kernel: [ 43.467923] Freezing user space
processes
May 6 16:00:04 feliu-vm kernel: [ 43.470294] Freezing user space
processes completed (elapsed 0.002 seconds)
May 6 16:00:04 feliu-vm kernel: [ 43.470299] OOM killer disabled.
May 6 16:00:04 feliu-vm kernel: [ 43.470301] Freezing remaining
freezable tasks
May 6 16:00:04 feliu-vm kernel: [ 43.471482] Freezing remaining
freezable tasks completed (elapsed 0.001 seconds)
May 6 16:00:04 feliu-vm kernel: [ 43.471495] printk: Suspending
console(s) (use no_console_suspend to debug)
May 6 16:00:04 feliu-vm kernel: [ 43.474034] virtio_net virtio0:
godeng virtio device freeze
May 6 16:00:04 feliu-vm kernel: [ 43.475714] virtio_net virtio0 ens3:
godfeng virtnet_freeze done
May 6 16:00:04 feliu-vm kernel: [ 43.475717] virtio_net virtio0:
godfeng VIRTIO_F_ADMIN_VQ not enabled
May 6 16:00:04 feliu-vm kernel: [ 43.475719] virtio_net virtio0:
godeng virtio device freeze done
........
May 6 16:00:04 feliu-vm kernel: [ 43.535382] smpboot: CPU 1 is now
offline
May 6 16:00:04 feliu-vm kernel: [ 43.537283] IRQ fixup: irq 1 move in
progress, old vector 32
May 6 16:00:04 feliu-vm kernel: [ 43.538504] smpboot: CPU 2 is now
offline
May 6 16:00:04 feliu-vm kernel: [ 43.541392] smpboot: CPU 3 is now
offline
......
May 6 16:00:04 feliu-vm kernel: [ 54.973285] smpboot: Booting Node 0
Processor 15 APIC 0xf
May 6 16:00:04 feliu-vm kernel: [ 54.975190] CPU15 is up
May 6 16:00:04 feliu-vm kernel: [ 54.976011] ACPI: PM: Waking up from
system sleep state S3
May 6 16:00:04 feliu-vm kernel: [ 54.986071] virtio_net virtio0:
godeng virtio device restore
May 6 16:00:04 feliu-vm kernel: [ 54.987563] virtio_net virtio0 ens3:
godfeng virtnet_restore done
May 6 16:00:04 feliu-vm kernel: [ 54.987635] virtio_net virtio0:
godfeng: virtio device restore done
......
May 6 16:00:04 feliu-vm kernel: [ 55.307221] ata8: SATA link down
(SStatus 0 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.442048] OOM killer enabled.
May 6 16:00:04 feliu-vm kernel: [ 55.442051] Restarting tasks ... done.
May 6 16:00:04 feliu-vm kernel: [ 55.443576] random: crng reseeded on
system resumption
May 6 16:00:04 feliu-vm kernel: [ 55.443582] PM: suspend exit
----------------------------------------------------------------------------
Attachment is the full kernel log. I think maybe it is some configration
error.
Thanks
Feng
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: sd 0:0:1:0: [sda] Synchronizing SCSI cache
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: PM: Some devices failed to suspend, or early wake event detected
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: OOM killer enabled.
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: Restarting tasks ... done.
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: random: crng reseeded on system resumption
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: PM: suspend exit
May 08 11:08:42 kernel-test-202405080702.c.ubuntu-catred.internal
kernel: PM: suspend entry (s2idle)
-- Boot 61828bc938b44fc68a8aeedc16a23a9d --
May 08 11:09:03 localhost kernel: Linux version 6.8.0-1007-gcp
(buildd@lcy02-amd64-079) (x86_64-linux-gnu-gcc-13 (Ubuntu
13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42)
#7-Ubuntu SMP Sat Apr 20 00:58:31 UTC 2024 (Ubuntu 6.8.0-1007.7-gcp 6.8.1)
May 08 11:09:03 localhost kernel: Command line:
BOOT_IMAGE=/vmlinuz-6.8.0-1007-gcp
root=PARTUUID=7a949935-6bf2-4cae-b404-803c95163572 ro
console=ttyS0,115200 panic=-1
2. The features the devices has:
catred@kernel-test-202405080702:~$ cat
/sys/bus/virtio/devices/virtio0/features
0110000000000000000000000000010000000000000000000000000000000000
catred@kernel-test-202405080702:~$ cat
/sys/bus/virtio/devices/virtio1/features
1110010110011001110000100000010000000000000000000000000000000000
catred@kernel-test-202405080702:~$ cat
/sys/bus/virtio/devices/virtio2/features
1110000000000000000000000000000000000000000000000000000000000000
catred@kernel-test-202405080702:~$ cat
/sys/bus/virtio/devices/virtio3/features
0000000000000000000000000000000000000000000000000000000000000000
Catherine
On Tue, May 7, 2024 at 11:34 PM Jason Wang <jasowang@xxxxxxxxxx
<mailto:jasowang@xxxxxxxxxx>> wrote:
On Sat, May 4, 2024 at 2:10 AM Joseph Salisbury
<joseph.salisbury@xxxxxxxxxxxxx
<mailto:joseph.salisbury@xxxxxxxxxxxxx>> wrote:
>
> Hi Feng,
>
> During testing, a kernel bug was identified with the suspend/resume
> functionality on instances running in a public cloud [0]. This
bug is a
> regression introduced in v6.8-rc1. After a kernel bisect, the
following
> commit was identified as the cause of the regression:
>
> fd27ef6b44be ("virtio-pci: Introduce admin virtqueue")
Have a quick glance at the patch it seems it should not damage the
freeze/restore as it should behave as in the past.
But I found something interesting:
1) assumes 1 admin vq which is not what spec said
2) special function for admin virtqueue during freeze/restore, but it
doesn't do anything special than del_vq()
3) lack real users but I guess e.g the destroy_avq() needs to be
synchronized with the one that is using admin virtqueue
>
> I was hoping to get your feedback, since you are the patch author. Do
> you think gathering any additional data will help diagnose this
issue?
Yes, please show us
1) the kernel log here.
2) the features that the device has like
/sys/bus/virtio/devices/virtio0/features
> This commit is depended upon by other virtio commits, so a revert
test
> is not really straight forward without reverting all the
dependencies.
> Any ideas you have would be greatly appreciated.
Thanks
>
>
> Thanks,
>
> Joe
>
> http://pad.lv/2063315 <http://pad.lv/2063315>
>
May 6 15:59:52 feliu-vm kernel: [ 43.446737] PM: suspend entry (deep)
May 6 16:00:04 feliu-vm kernel: [ 43.467640] Filesystems sync: 0.020 seconds
May 6 16:00:04 feliu-vm kernel: [ 43.467923] Freezing user space processes
May 6 16:00:04 feliu-vm kernel: [ 43.470294] Freezing user space processes completed (elapsed 0.002 seconds)
May 6 16:00:04 feliu-vm kernel: [ 43.470299] OOM killer disabled.
May 6 16:00:04 feliu-vm kernel: [ 43.470301] Freezing remaining freezable tasks
May 6 16:00:04 feliu-vm kernel: [ 43.471482] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
May 6 16:00:04 feliu-vm kernel: [ 43.471495] printk: Suspending console(s) (use no_console_suspend to debug)
May 6 16:00:04 feliu-vm kernel: [ 43.474034] virtio_net virtio0: godeng virtio device freeze
May 6 16:00:04 feliu-vm kernel: [ 43.475714] virtio_net virtio0 ens3: godfeng virtnet_freeze done
May 6 16:00:04 feliu-vm kernel: [ 43.475717] virtio_net virtio0: godfeng VIRTIO_F_ADMIN_VQ not enabled
May 6 16:00:04 feliu-vm kernel: [ 43.475719] virtio_net virtio0: godeng virtio device freeze done
May 6 16:00:04 feliu-vm kernel: [ 43.528538] sd 3:0:0:0: [sdb] Synchronizing SCSI cache
May 6 16:00:04 feliu-vm kernel: [ 43.528599] sd 2:0:0:0: [sda] Synchronizing SCSI cache
May 6 16:00:04 feliu-vm kernel: [ 43.529288] ata4.00: Entering standby power mode
May 6 16:00:04 feliu-vm kernel: [ 43.529364] ata3.00: Entering standby power mode
May 6 16:00:04 feliu-vm kernel: [ 43.532432] ACPI: PM: Preparing to enter system sleep state S3
May 6 16:00:04 feliu-vm kernel: [ 43.532534] ACPI: PM: Saving platform NVS memory
May 6 16:00:04 feliu-vm kernel: [ 43.532570] Disabling non-boot CPUs ...
May 6 16:00:04 feliu-vm kernel: [ 43.535382] smpboot: CPU 1 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.537283] IRQ fixup: irq 1 move in progress, old vector 32
May 6 16:00:04 feliu-vm kernel: [ 43.538504] smpboot: CPU 2 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.541392] smpboot: CPU 3 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.544464] smpboot: CPU 4 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.547296] smpboot: CPU 5 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.550076] smpboot: CPU 6 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.552660] smpboot: CPU 7 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.555278] smpboot: CPU 8 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.557693] smpboot: CPU 9 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.560292] smpboot: CPU 10 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.562624] smpboot: CPU 11 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.565185] smpboot: CPU 12 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.567496] smpboot: CPU 13 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.570153] smpboot: CPU 14 is now offline
May 6 16:00:04 feliu-vm kernel: [ 43.572965] smpboot: CPU 15 is now offline
May 6 16:00:04 feliu-vm kernel: [ 54.947312] unchecked MSR access error: WRMSR to 0xc0011029 (tried to write 0x0000000000000000) at rIP: 0xffffffffac2b4c48 (native_write_msr+0x8/0x30)
May 6 16:00:04 feliu-vm kernel: [ 54.947333] Call Trace:
May 6 16:00:04 feliu-vm kernel: [ 54.947337] <TASK>
May 6 16:00:04 feliu-vm kernel: [ 54.947349] ? show_stack_regs+0x26/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947355] ? ex_handler_msr+0x10f/0x180
May 6 16:00:04 feliu-vm kernel: [ 54.947359] ? search_extable+0x2b/0x40
May 6 16:00:04 feliu-vm kernel: [ 54.947365] ? fixup_exception+0x315/0x380
May 6 16:00:04 feliu-vm kernel: [ 54.947369] ? exc_general_protection+0x152/0x480
May 6 16:00:04 feliu-vm kernel: [ 54.947373] ? acpi_hw_read_port+0xec/0x100
May 6 16:00:04 feliu-vm kernel: [ 54.947382] ? asm_exc_general_protection+0x2b/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947397] ? native_write_msr+0x8/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947401] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947405] ? restore_processor_state+0x2bf/0x350
May 6 16:00:04 feliu-vm kernel: [ 54.947411] x86_acpi_suspend_lowlevel+0xf7/0x160
May 6 16:00:04 feliu-vm kernel: [ 54.947416] acpi_suspend_enter+0x5a/0x1f0
May 6 16:00:04 feliu-vm kernel: [ 54.947421] ? _raw_spin_unlock_irqrestore+0x12/0x40
May 6 16:00:04 feliu-vm kernel: [ 54.947425] suspend_devices_and_enter+0x759/0x8e0
May 6 16:00:04 feliu-vm kernel: [ 54.947432] pm_suspend+0x23e/0x5e0
May 6 16:00:04 feliu-vm kernel: [ 54.947435] state_store+0x85/0xf0
May 6 16:00:04 feliu-vm kernel: [ 54.947440] kobj_attr_store+0x16/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947444] sysfs_kf_write+0x42/0x50
May 6 16:00:04 feliu-vm kernel: [ 54.947448] kernfs_fop_write_iter+0x143/0x1d0
May 6 16:00:04 feliu-vm kernel: [ 54.947453] vfs_write+0x311/0x430
May 6 16:00:04 feliu-vm kernel: [ 54.947461] ksys_write+0x6b/0xf0
May 6 16:00:04 feliu-vm kernel: [ 54.947466] __x64_sys_write+0x1d/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947469] x64_sys_call+0x1681/0x20c0
May 6 16:00:04 feliu-vm kernel: [ 54.947474] do_syscall_64+0x71/0x110
May 6 16:00:04 feliu-vm kernel: [ 54.947478] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947482] ? kmem_cache_free+0x3bb/0x3f0
May 6 16:00:04 feliu-vm kernel: [ 54.947486] ? __fput+0x16d/0x2f0
May 6 16:00:04 feliu-vm kernel: [ 54.947491] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947494] ? __fput+0x16d/0x2f0
May 6 16:00:04 feliu-vm kernel: [ 54.947498] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947502] ? __fput_sync+0x1e/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947505] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947508] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947511] ? syscall_exit_to_user_mode+0x92/0x1c0
May 6 16:00:04 feliu-vm kernel: [ 54.947515] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947518] ? do_syscall_64+0x7d/0x110
May 6 16:00:04 feliu-vm kernel: [ 54.947521] ? set_ptes.constprop.0+0x2b/0x90
May 6 16:00:04 feliu-vm kernel: [ 54.947526] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947529] ? next_uptodate_folio+0x9f/0x310
May 6 16:00:04 feliu-vm kernel: [ 54.947534] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947537] ? _raw_spin_unlock+0x12/0x30
May 6 16:00:04 feliu-vm kernel: [ 54.947540] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947543] ? filemap_map_pages+0x4c3/0x5c0
May 6 16:00:04 feliu-vm kernel: [ 54.947547] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947550] ? __pte_offset_map+0x20/0x190
May 6 16:00:04 feliu-vm kernel: [ 54.947557] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947560] ? do_fault+0x252/0x4a0
May 6 16:00:04 feliu-vm kernel: [ 54.947564] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947567] ? __handle_mm_fault+0x8b0/0xeb0
May 6 16:00:04 feliu-vm kernel: [ 54.947570] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947576] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947579] ? __task_pid_nr_ns+0x70/0xc0
May 6 16:00:04 feliu-vm kernel: [ 54.947583] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947586] ? syscall_exit_to_user_mode+0x92/0x1c0
May 6 16:00:04 feliu-vm kernel: [ 54.947590] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947593] ? do_syscall_64+0x7d/0x110
May 6 16:00:04 feliu-vm kernel: [ 54.947596] ? handle_mm_fault+0x190/0x2f0
May 6 16:00:04 feliu-vm kernel: [ 54.947600] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947603] ? do_user_addr_fault+0x357/0x6b0
May 6 16:00:04 feliu-vm kernel: [ 54.947606] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947609] ? irqentry_exit_to_user_mode+0x63/0x180
May 6 16:00:04 feliu-vm kernel: [ 54.947613] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947616] ? irqentry_exit+0x3f/0x50
May 6 16:00:04 feliu-vm kernel: [ 54.947620] ? srso_return_thunk+0x5/0x5f
May 6 16:00:04 feliu-vm kernel: [ 54.947623] ? exc_page_fault+0x8d/0x190
May 6 16:00:04 feliu-vm kernel: [ 54.947627] entry_SYSCALL_64_after_hwframe+0x76/0x7e
May 6 16:00:04 feliu-vm kernel: [ 54.947631] RIP: 0033:0x7f0ebcd14887
May 6 16:00:04 feliu-vm kernel: [ 54.947641] Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
May 6 16:00:04 feliu-vm kernel: [ 54.947644] RSP: 002b:00007ffee36a71d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
May 6 16:00:04 feliu-vm kernel: [ 54.947647] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f0ebcd14887
May 6 16:00:04 feliu-vm kernel: [ 54.947649] RDX: 0000000000000004 RSI: 00007ffee36a7290 RDI: 0000000000000004
May 6 16:00:04 feliu-vm kernel: [ 54.947651] RBP: 00007ffee36a7290 R08: 0000000000000004 R09: 000000007fffffff
May 6 16:00:04 feliu-vm kernel: [ 54.947653] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
May 6 16:00:04 feliu-vm kernel: [ 54.947654] R13: 000055f0fdddc2d0 R14: 00007f0ebce16a00 R15: 0000000000000004
May 6 16:00:04 feliu-vm kernel: [ 54.947661] </TASK>
May 6 16:00:04 feliu-vm kernel: [ 54.947665] ACPI: PM: Low-level resume complete
May 6 16:00:04 feliu-vm kernel: [ 54.947715] ACPI: PM: Restoring platform NVS memory
May 6 16:00:04 feliu-vm kernel: [ 54.949557] Enabling non-boot CPUs ...
May 6 16:00:04 feliu-vm kernel: [ 54.949649] smpboot: Booting Node 0 Processor 1 APIC 0x1
May 6 16:00:04 feliu-vm kernel: [ 54.951003] CPU1 is up
May 6 16:00:04 feliu-vm kernel: [ 54.951050] smpboot: Booting Node 0 Processor 2 APIC 0x2
May 6 16:00:04 feliu-vm kernel: [ 54.953425] CPU2 is up
May 6 16:00:04 feliu-vm kernel: [ 54.953478] smpboot: Booting Node 0 Processor 3 APIC 0x3
May 6 16:00:04 feliu-vm kernel: [ 54.954602] CPU3 is up
May 6 16:00:04 feliu-vm kernel: [ 54.954654] smpboot: Booting Node 0 Processor 4 APIC 0x4
May 6 16:00:04 feliu-vm kernel: [ 54.956254] CPU4 is up
May 6 16:00:04 feliu-vm kernel: [ 54.956309] smpboot: Booting Node 0 Processor 5 APIC 0x5
May 6 16:00:04 feliu-vm kernel: [ 54.957780] CPU5 is up
May 6 16:00:04 feliu-vm kernel: [ 54.957835] smpboot: Booting Node 0 Processor 6 APIC 0x6
May 6 16:00:04 feliu-vm kernel: [ 54.959263] CPU6 is up
May 6 16:00:04 feliu-vm kernel: [ 54.959312] smpboot: Booting Node 0 Processor 7 APIC 0x7
May 6 16:00:04 feliu-vm kernel: [ 54.960734] CPU7 is up
May 6 16:00:04 feliu-vm kernel: [ 54.960788] smpboot: Booting Node 0 Processor 8 APIC 0x8
May 6 16:00:04 feliu-vm kernel: [ 54.962273] CPU8 is up
May 6 16:00:04 feliu-vm kernel: [ 54.962328] smpboot: Booting Node 0 Processor 9 APIC 0x9
May 6 16:00:04 feliu-vm kernel: [ 54.963692] CPU9 is up
May 6 16:00:04 feliu-vm kernel: [ 54.963739] smpboot: Booting Node 0 Processor 10 APIC 0xa
May 6 16:00:04 feliu-vm kernel: [ 54.965708] CPU10 is up
May 6 16:00:04 feliu-vm kernel: [ 54.965769] smpboot: Booting Node 0 Processor 11 APIC 0xb
May 6 16:00:04 feliu-vm kernel: [ 54.967317] CPU11 is up
May 6 16:00:04 feliu-vm kernel: [ 54.967352] smpboot: Booting Node 0 Processor 12 APIC 0xc
May 6 16:00:04 feliu-vm kernel: [ 54.968943] CPU12 is up
May 6 16:00:04 feliu-vm kernel: [ 54.968991] smpboot: Booting Node 0 Processor 13 APIC 0xd
May 6 16:00:04 feliu-vm kernel: [ 54.970551] CPU13 is up
May 6 16:00:04 feliu-vm kernel: [ 54.970602] smpboot: Booting Node 0 Processor 14 APIC 0xe
May 6 16:00:04 feliu-vm kernel: [ 54.973225] CPU14 is up
May 6 16:00:04 feliu-vm kernel: [ 54.973285] smpboot: Booting Node 0 Processor 15 APIC 0xf
May 6 16:00:04 feliu-vm kernel: [ 54.975190] CPU15 is up
May 6 16:00:04 feliu-vm kernel: [ 54.976011] ACPI: PM: Waking up from system sleep state S3
May 6 16:00:04 feliu-vm kernel: [ 54.986071] virtio_net virtio0: godeng virtio device restore
May 6 16:00:04 feliu-vm kernel: [ 54.987563] virtio_net virtio0 ens3: godfeng virtnet_restore done
May 6 16:00:04 feliu-vm kernel: [ 54.987635] virtio_net virtio0: godfeng: virtio device restore done
May 6 16:00:04 feliu-vm kernel: [ 54.988137] usb usb1: root hub lost power or was reset
May 6 16:00:04 feliu-vm kernel: [ 54.988640] pci 0000:00:01.0: PIIX3: Enabling Passive Release
May 6 16:00:04 feliu-vm kernel: [ 54.989846] usb usb3: root hub lost power or was reset
May 6 16:00:04 feliu-vm kernel: [ 54.989908] usb usb4: root hub lost power or was reset
May 6 16:00:04 feliu-vm kernel: [ 54.989961] usb usb2: root hub lost power or was reset
May 6 16:00:04 feliu-vm kernel: [ 55.272222] usb 2-1: reset high-speed USB device number 2 using ehci-pci
May 6 16:00:04 feliu-vm kernel: [ 55.302948] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.303093] ata5: SATA link down (SStatus 0 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.303266] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.303367] sd 3:0:0:0: [sdb] Starting disk
May 6 16:00:04 feliu-vm kernel: [ 55.303434] sd 2:0:0:0: [sda] Starting disk
May 6 16:00:04 feliu-vm kernel: [ 55.303516] ata4.00: configured for UDMA/100
May 6 16:00:04 feliu-vm kernel: [ 55.303563] ata3.00: configured for UDMA/100
May 6 16:00:04 feliu-vm kernel: [ 55.306912] ata6: SATA link down (SStatus 0 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.307071] ata7: SATA link down (SStatus 0 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.307221] ata8: SATA link down (SStatus 0 SControl 300)
May 6 16:00:04 feliu-vm kernel: [ 55.442048] OOM killer enabled.
May 6 16:00:04 feliu-vm kernel: [ 55.442051] Restarting tasks ... done.
May 6 16:00:04 feliu-vm kernel: [ 55.443576] random: crng reseeded on system resumption
May 6 16:00:04 feliu-vm kernel: [ 55.443582] PM: suspend exit
