I got new trace, attached as console-1566240256.588497319.log Maybe this is new or just something not seen before. After I have applied the patch, it seems issues with Call Traces starting with "zswap_writeback_entry+0x50" are not happening any more. Previously they were about 40% of issues. [ 590.980777] BUG: kernel NULL pointer dereference, address: 0000000000000008 [ 590.985408] #PF: supervisor read access in kernel mode [ 590.989346] #PF: error_code(0x0000) - not-present page [ 590.993317] PGD 0 P4D 0 [ 590.995272] Oops: 0000 [#1] SMP PTI [ 590.997963] CPU: 1 PID: 41 Comm: kworker/u8:1 Not tainted 5.3.0-rc5+ #71 [ 591.003103] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 591.009620] Workqueue: zswap1 compact_page_work [ 591.013084] RIP: 0010:__list_add_valid+0x3/0x40 [ 591.016570] Code: f4 ff ff ff e9 3a ff ff ff 49 c7 07 00 00 00 00 41 c7 47 08 00 00 00 00 e9 66 ff ff ff e8 d5 f4 b6 ff 90 90 90 90 90 49 89 d0 <48> 8b 52 08 48 39 f2 0f 85 7c 00 00 00 4c 8b 0a 4d 39 c1 0f 85 98 [ 591.030935] RSP: 0000:ffffa6f1c0177df8 EFLAGS: 00010206 [ 591.034940] RAX: 00000000000003c0 RBX: ffff8d046b5f7000 RCX: 8888888888888889 [ 591.040425] RDX: 0000000000000000 RSI: ffffc6f1bf8012e8 RDI: ffff8d046b5f7000 [ 591.045868] RBP: ffffc6f1bf8012e8 R08: 0000000000000000 R09: 0000000000000001 [ 591.051305] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 [ 591.056762] R13: ffff8d0475f5ba00 R14: ffff8d0475f5ba08 R15: ffff8d046b5f7010 [ 591.062213] FS: 0000000000000000(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 591.068390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 591.072756] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 591.078217] Call Trace: [ 591.080093] do_compact_page+0x31e/0x430 [ 591.083124] process_one_work+0x272/0x5a0 [ 591.086183] worker_thread+0x50/0x3b0 [ 591.089003] kthread+0x108/0x140 [ 591.091488] ? process_one_work+0x5a0/0x5a0 [ 591.094695] ? kthread_park+0x80/0x80 [ 591.097510] ret_from_fork+0x3a/0x50 [ 591.100275] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 591.141313] CR2: 0000000000000008 [ 591.143848] ---[ end trace adaa9348e173db62 ]---
Fedora 30 (Thirty) Kernel 5.3.0-rc5+ on an x86_64 (ttyS0) localhost login: [ 590.980777] BUG: kernel NULL pointer dereference, address: 0000000000000008 [ 590.985408] #PF: supervisor read access in kernel mode [ 590.989346] #PF: error_code(0x0000) - not-present page [ 590.993317] PGD 0 P4D 0 [ 590.995272] Oops: 0000 [#1] SMP PTI [ 590.997963] CPU: 1 PID: 41 Comm: kworker/u8:1 Not tainted 5.3.0-rc5+ #71 [ 591.003103] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 591.009620] Workqueue: zswap1 compact_page_work [ 591.013084] RIP: 0010:__list_add_valid+0x3/0x40 [ 591.016570] Code: f4 ff ff ff e9 3a ff ff ff 49 c7 07 00 00 00 00 41 c7 47 08 00 00 00 00 e9 66 ff ff ff e8 d5 f4 b6 ff 90 90 90 90 90 49 89 d0 <48> 8b 52 08 48 39 f2 0f 85 7c 00 00 00 4c 8b 0a 4d 39 c1 0f 85 98 [ 591.030935] RSP: 0000:ffffa6f1c0177df8 EFLAGS: 00010206 [ 591.034940] RAX: 00000000000003c0 RBX: ffff8d046b5f7000 RCX: 8888888888888889 [ 591.040425] RDX: 0000000000000000 RSI: ffffc6f1bf8012e8 RDI: ffff8d046b5f7000 [ 591.045868] RBP: ffffc6f1bf8012e8 R08: 0000000000000000 R09: 0000000000000001 [ 591.051305] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 [ 591.056762] R13: ffff8d0475f5ba00 R14: ffff8d0475f5ba08 R15: ffff8d046b5f7010 [ 591.062213] FS: 0000000000000000(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 591.068390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 591.072756] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 591.078217] Call Trace: [ 591.080093] do_compact_page+0x31e/0x430 [ 591.083124] process_one_work+0x272/0x5a0 [ 591.086183] worker_thread+0x50/0x3b0 [ 591.089003] kthread+0x108/0x140 [ 591.091488] ? process_one_work+0x5a0/0x5a0 [ 591.094695] ? kthread_park+0x80/0x80 [ 591.097510] ret_from_fork+0x3a/0x50 [ 591.100275] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 591.141313] CR2: 0000000000000008 [ 591.143848] ---[ end trace adaa9348e173db62 ]--- [ 591.147371] RIP: 0010:__list_add_valid+0x3/0x40 [ 591.150853] Code: f4 ff ff ff e9 3a ff ff ff 49 c7 07 00 00 00 00 41 c7 47 08 00 00 00 00 e9 66 ff ff ff e8 d5 f4 b6 ff 90 90 90 90 90 49 89 d0 <48> 8b 52 08 48 39 f2 0f 85 7c 00 00 00 4c 8b 0a 4d 39 c1 0f 85 98 [ 591.165056] RSP: 0000:ffffa6f1c0177df8 EFLAGS: 00010206 [ 591.169041] RAX: 00000000000003c0 RBX: ffff8d046b5f7000 RCX: 8888888888888889 [ 591.174461] RDX: 0000000000000000 RSI: ffffc6f1bf8012e8 RDI: ffff8d046b5f7000 [ 591.179909] RBP: ffffc6f1bf8012e8 R08: 0000000000000000 R09: 0000000000000001 [ 591.185346] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 [ 591.188460] R13: ffff8d0475f5ba00 R14: ffff8d0475f5ba08 R15: ffff8d046b5f7010 [ 591.189970] FS: 0000000000000000(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 591.191584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 591.194392] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 591.197692] BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:38 [ 591.201603] in_atomic(): 1, irqs_disabled(): 1, pid: 41, name: kworker/u8:1 [ 591.204828] INFO: lockdep is turned off. [ 591.206972] irq event stamp: 15078918 [ 591.209026] hardirqs last enabled at (15078917): [<ffffffff909d6b09>] _raw_spin_unlock_irq+0x29/0x40 [ 591.213085] hardirqs last disabled at (15078918): [<ffffffff90001c8a>] trace_hardirqs_off_thunk+0x1a/0x20 [ 591.217256] softirqs last enabled at (15078804): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 591.221123] softirqs last disabled at (15078797): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 591.224821] CPU: 1 PID: 41 Comm: kworker/u8:1 Tainted: G D 5.3.0-rc5+ #71 [ 591.228464] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 591.232254] Workqueue: zswap1 compact_page_work [ 591.234603] Call Trace: [ 591.236203] dump_stack+0x67/0x90 [ 591.238103] ___might_sleep.cold+0x9f/0xaf [ 591.240274] exit_signals+0x30/0x330 [ 591.242250] do_exit+0xcb/0xcd0 [ 591.244072] ? kthread+0x108/0x140 [ 591.245976] rewind_stack_do_exit+0x17/0x20 [ 591.248183] note: kworker/u8:1[41] exited with preempt_count 3 [ 611.227034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [stress:1826] [ 611.232209] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 611.269976] irq event stamp: 132973063 [ 611.273164] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 611.280262] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 611.287086] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 611.293450] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 611.299524] CPU: 0 PID: 1826 Comm: stress Tainted: G D W 5.3.0-rc5+ #71 [ 611.305202] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 611.311360] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 611.315592] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 611.328995] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 611.334576] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 611.339870] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 611.345144] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 611.350032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 611.350408] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 611.352759] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 611.356994] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 611.356998] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 611.373083] irq event stamp: 136226147 [ 611.373087] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 611.377506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 611.377507] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 611.380160] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 611.380163] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 611.382780] Call Trace: [ 611.385643] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 611.385645] CPU: 2 PID: 1824 Comm: stress Tainted: G D W 5.3.0-rc5+ #71 [ 611.389415] do_raw_spin_lock+0xab/0xb0 [ 611.391844] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 611.391847] RIP: 0010:queued_spin_lock_slowpath+0x44/0x1e0 [ 611.397180] _raw_spin_lock+0x63/0x80 [ 611.399990] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 c3 f6 [ 611.399991] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 611.402089] z3fold_zpool_shrink+0x2d3/0x570 [ 611.404809] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 611.404810] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 611.409564] ? sched_clock_cpu+0xc/0xc0 [ 611.411258] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 611.411260] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 611.416372] zswap_frontswap_store+0x424/0x7c1 [ 611.418465] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 611.418468] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 611.421233] __frontswap_store+0xc4/0x162 [ 611.427237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 611.427239] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 611.431916] swap_writepage+0x39/0x70 [ 611.433794] Call Trace: [ 611.438247] pageout.isra.0+0x12c/0x5d0 [ 611.440776] do_raw_spin_lock+0xab/0xb0 [ 611.443596] shrink_page_list+0x1124/0x1830 [ 611.446104] _raw_spin_lock+0x63/0x80 [ 611.450514] shrink_inactive_list+0x1da/0x460 [ 611.452389] __swp_swapcount+0xb9/0xf0 [ 611.456818] shrink_node_memcg+0x202/0x770 [ 611.459504] __read_swap_cache_async+0xc0/0x3e0 [ 611.462315] ? sched_clock_cpu+0xc/0xc0 [ 611.464463] swap_cluster_readahead+0x184/0x330 [ 611.468864] shrink_node+0xdf/0x490 [ 611.470543] ? find_held_lock+0x32/0x90 [ 611.472541] do_try_to_free_pages+0xdb/0x3c0 [ 611.474242] swapin_readahead+0x2b4/0x4e0 [ 611.476966] try_to_free_pages+0x112/0x2e0 [ 611.478714] ? sched_clock_cpu+0xc/0xc0 [ 611.481319] __alloc_pages_slowpath+0x422/0x1000 [ 611.483079] do_swap_page+0x3ac/0xc20 [ 611.485688] ? __lock_acquire+0x247/0x1900 [ 611.487381] __handle_mm_fault+0x8da/0x1900 [ 611.490365] __alloc_pages_nodemask+0x37f/0x400 [ 611.491981] handle_mm_fault+0x159/0x340 [ 611.494964] alloc_pages_vma+0xcc/0x170 [ 611.496486] do_user_addr_fault+0x1fe/0x480 [ 611.499168] __read_swap_cache_async+0x1e9/0x3e0 [ 611.500824] do_page_fault+0x31/0x210 [ 611.503519] swap_cluster_readahead+0x184/0x330 [ 611.505101] page_fault+0x3e/0x50 [ 611.507621] ? find_held_lock+0x32/0x90 [ 611.509306] RIP: 0033:0x563ef1681298 [ 611.511784] swapin_readahead+0x2b4/0x4e0 [ 611.513322] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 611.516042] ? sched_clock_cpu+0xc/0xc0 [ 611.517658] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 611.520258] do_swap_page+0x3ac/0xc20 [ 611.521703] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 611.521704] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 611.524384] __handle_mm_fault+0x8da/0x1900 [ 611.525987] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 611.525988] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 611.528365] handle_mm_fault+0x159/0x340 [ 611.529929] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 611.583430] do_user_addr_fault+0x1fe/0x480 [ 611.586043] do_page_fault+0x31/0x210 [ 611.588430] page_fault+0x3e/0x50 [ 611.590629] RIP: 0033:0x563ef1681298 [ 611.592904] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 611.603434] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 611.606659] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 611.610795] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 611.614922] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 611.619062] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 611.623124] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 615.289024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [stress:1823] [ 615.291208] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 615.306341] irq event stamp: 156337142 [ 615.307849] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 615.310678] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 615.313272] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 615.315925] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 615.318478] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 615.320904] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 615.323525] RIP: 0010:queued_spin_lock_slowpath+0x124/0x1e0 [ 615.325484] Code: 00 89 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 [ 615.331156] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 615.333615] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 615.335986] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 615.338353] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 615.340724] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 615.343103] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 615.345483] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 615.348086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 615.350148] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 615.352517] Call Trace: [ 615.353820] do_raw_spin_lock+0xab/0xb0 [ 615.355443] _raw_spin_lock+0x63/0x80 [ 615.357030] __swap_duplicate+0x163/0x220 [ 615.358676] __read_swap_cache_async+0x1f9/0x3e0 [ 615.360457] swap_cluster_readahead+0x184/0x330 [ 615.362213] ? find_held_lock+0x32/0x90 [ 615.363804] swapin_readahead+0x2b4/0x4e0 [ 615.365435] ? sched_clock_cpu+0xc/0xc0 [ 615.367021] do_swap_page+0x3ac/0xc20 [ 615.368554] __handle_mm_fault+0x8da/0x1900 [ 615.370205] handle_mm_fault+0x159/0x340 [ 615.371787] do_user_addr_fault+0x1fe/0x480 [ 615.373435] do_page_fault+0x31/0x210 [ 615.374961] page_fault+0x3e/0x50 [ 615.376398] RIP: 0033:0x563ef1681298 [ 615.377870] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 615.383431] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 615.385301] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 615.387602] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 615.389902] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 615.392205] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 615.394503] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 639.226970] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [stress:1826] [ 639.231900] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 639.269518] irq event stamp: 132973063 [ 639.272603] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 639.279634] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 639.286408] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 639.292720] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 639.298745] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 639.304383] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 639.310512] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 639.314695] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 639.328070] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 639.333630] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 639.338906] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 639.344171] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 639.349438] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 639.349969] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 639.354691] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 639.356981] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 639.361127] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 639.376931] irq event stamp: 136226147 [ 639.376935] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 639.381597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 639.381598] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 639.383247] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 639.383249] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 639.388384] Call Trace: [ 639.390506] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 639.390508] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 639.394727] do_raw_spin_lock+0xab/0xb0 [ 639.397604] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 639.397607] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 639.402678] _raw_spin_lock+0x63/0x80 [ 639.404087] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 639.408961] z3fold_zpool_shrink+0x2d3/0x570 [ 639.411585] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 639.414296] ? sched_clock_cpu+0xc/0xc0 [ 639.417106] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 639.417107] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 639.420621] zswap_frontswap_store+0x424/0x7c1 [ 639.422313] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 639.422314] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 639.433199] __frontswap_store+0xc4/0x162 [ 639.435066] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 639.435069] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 639.439818] swap_writepage+0x39/0x70 [ 639.441607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 639.441608] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 639.446149] pageout.isra.0+0x12c/0x5d0 [ 639.448667] Call Trace: [ 639.451770] shrink_page_list+0x1124/0x1830 [ 639.454268] do_raw_spin_lock+0xab/0xb0 [ 639.458732] shrink_inactive_list+0x1da/0x460 [ 639.460489] _raw_spin_lock+0x63/0x80 [ 639.464887] shrink_node_memcg+0x202/0x770 [ 639.467562] __swp_swapcount+0xb9/0xf0 [ 639.470227] ? sched_clock_cpu+0xc/0xc0 [ 639.472380] __read_swap_cache_async+0xc0/0x3e0 [ 639.476831] shrink_node+0xdf/0x490 [ 639.478548] swap_cluster_readahead+0x184/0x330 [ 639.480579] do_try_to_free_pages+0xdb/0x3c0 [ 639.482351] ? find_held_lock+0x32/0x90 [ 639.485082] try_to_free_pages+0x112/0x2e0 [ 639.486864] swapin_readahead+0x2b4/0x4e0 [ 639.489467] __alloc_pages_slowpath+0x422/0x1000 [ 639.491165] ? sched_clock_cpu+0xc/0xc0 [ 639.493789] ? __lock_acquire+0x247/0x1900 [ 639.495420] do_swap_page+0x3ac/0xc20 [ 639.498409] __alloc_pages_nodemask+0x37f/0x400 [ 639.499949] __handle_mm_fault+0x8da/0x1900 [ 639.502953] alloc_pages_vma+0xcc/0x170 [ 639.504655] handle_mm_fault+0x159/0x340 [ 639.507291] __read_swap_cache_async+0x1e9/0x3e0 [ 639.508913] do_user_addr_fault+0x1fe/0x480 [ 639.511611] swap_cluster_readahead+0x184/0x330 [ 639.513317] do_page_fault+0x31/0x210 [ 639.515855] ? find_held_lock+0x32/0x90 [ 639.517422] page_fault+0x3e/0x50 [ 639.519847] swapin_readahead+0x2b4/0x4e0 [ 639.521486] RIP: 0033:0x563ef1681298 [ 639.524229] ? sched_clock_cpu+0xc/0xc0 [ 639.525689] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 639.528274] do_swap_page+0x3ac/0xc20 [ 639.529902] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 639.532553] __handle_mm_fault+0x8da/0x1900 [ 639.534148] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 639.534149] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 639.536549] handle_mm_fault+0x159/0x340 [ 639.537961] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 639.537963] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 639.540169] do_user_addr_fault+0x1fe/0x480 [ 639.541607] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 639.590306] do_page_fault+0x31/0x210 [ 639.592686] page_fault+0x3e/0x50 [ 639.594925] RIP: 0033:0x563ef1681298 [ 639.597255] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 639.607850] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 639.611072] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 639.615207] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 639.619342] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 639.623524] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 639.627647] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 643.288961] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [stress:1823] [ 643.294003] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 643.330725] irq event stamp: 156337142 [ 643.333740] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 643.340410] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 643.346424] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 643.352607] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 643.358510] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 643.364055] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 643.370095] RIP: 0010:queued_spin_lock_slowpath+0x124/0x1e0 [ 643.374298] Code: 00 89 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 [ 643.387497] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 643.392994] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 643.397582] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 643.402816] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 643.408164] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 643.413378] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 643.418650] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 643.424627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 643.429088] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 643.434428] Call Trace: [ 643.436757] do_raw_spin_lock+0xab/0xb0 [ 643.439974] _raw_spin_lock+0x63/0x80 [ 643.443081] __swap_duplicate+0x163/0x220 [ 643.446404] __read_swap_cache_async+0x1f9/0x3e0 [ 643.450097] swap_cluster_readahead+0x184/0x330 [ 643.453650] ? find_held_lock+0x32/0x90 [ 643.456839] swapin_readahead+0x2b4/0x4e0 [ 643.460143] ? sched_clock_cpu+0xc/0xc0 [ 643.463337] do_swap_page+0x3ac/0xc20 [ 643.466397] __handle_mm_fault+0x8da/0x1900 [ 643.469781] handle_mm_fault+0x159/0x340 [ 643.472990] do_user_addr_fault+0x1fe/0x480 [ 643.476372] do_page_fault+0x31/0x210 [ 643.479337] page_fault+0x3e/0x50 [ 643.482127] RIP: 0033:0x563ef1681298 [ 643.485081] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 643.498228] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 643.502261] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 643.507428] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 643.512676] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 643.517931] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 643.523206] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 655.945932] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 655.950436] rcu: 3-...!: (35 ticks this GP) idle=1b6/1/0x4000000000000000 softirq=510653/510653 fqs=9 [ 655.957039] (detected by 1, t=65013 jiffies, g=249541, q=159) [ 655.961376] Sending NMI from CPU 1 to CPUs 3: [ 655.965800] NMI backtrace for cpu 3 [ 655.965800] CPU: 3 PID: 1822 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 655.965801] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 655.965801] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 655.965801] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 655.965802] RSP: 0018:ffffa6f1c08efaf8 EFLAGS: 00000002 [ 655.965802] RAX: 0000000000000101 RBX: ffff8d0475f5ba08 RCX: 8888888888888889 [ 655.965803] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d0475f5ba08 [ 655.965803] RBP: ffff8d0475f5ba08 R08: 000000899932f941 R09: 0000000000000000 [ 655.965803] R10: 0000000000000006 R11: 0000000000000000 R12: ffff8d0475f5ba20 [ 655.965804] R13: ffff8d04527e6000 R14: ffff8d04527e6010 R15: 0000000000000000 [ 655.965804] FS: 00007f35ba5bb740(0000) GS:ffff8d047ea00000(0000) knlGS:0000000000000000 [ 655.965804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 655.965805] CR2: 00007f35ba77e2c8 CR3: 000000002ed50005 CR4: 0000000000160ee0 [ 655.965805] Call Trace: [ 655.965805] do_raw_spin_lock+0xab/0xb0 [ 655.965805] _raw_spin_lock+0x63/0x80 [ 655.965806] z3fold_zpool_free+0x29d/0x323 [ 655.965806] zswap_free_entry+0x43/0x50 [ 655.965806] zswap_frontswap_invalidate_page+0x8c/0x90 [ 655.965806] __frontswap_invalidate_page+0x56/0x90 [ 655.965807] swap_range_free+0xb2/0xd0 [ 655.965807] swapcache_free_entries+0x128/0x1a0 [ 655.965807] free_swap_slot+0xd5/0xf0 [ 655.965807] __swap_entry_free.constprop.0+0x8c/0xa0 [ 655.965808] free_swap_and_cache+0x35/0x70 [ 655.965808] unmap_page_range+0x4c8/0xd00 [ 655.965808] unmap_vmas+0x70/0xd0 [ 655.965808] unmap_region+0xa8/0x110 [ 655.965809] __do_munmap+0x297/0x460 [ 655.965809] __vm_munmap+0x6a/0xc0 [ 655.965809] __x64_sys_munmap+0x28/0x30 [ 655.965809] do_syscall_64+0x5a/0x220 [ 655.965810] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 655.965810] RIP: 0033:0x7f35ba6b41eb [ 655.965810] Code: 8b 15 a1 9c 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb 89 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6d 9c 0c 00 f7 d8 64 89 01 48 [ 655.965811] RSP: 002b:00007ffcd50ca7a8 EFLAGS: 00000202 ORIG_RAX: 000000000000000b [ 655.965811] RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 00007f35ba6b41eb [ 655.965812] RDX: 000000000000000f RSI: 000000000b86f000 RDI: 00007f35aed4c000 [ 655.965812] RBP: 00007f35aed4c010 R08: 00007f35aed4c000 R09: 0000000000000000 [ 655.965812] R10: 00007f35ba5ba010 R11: 0000000000000202 R12: 0000563ef1683004 [ 655.965813] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 655.965815] rcu: rcu_sched kthread starved for 64985 jiffies! g249541 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 656.156681] rcu: RCU grace-period kthread stack dump: [ 656.160435] rcu_sched R running task 14280 10 2 0x80004000 [ 656.165442] Call Trace: [ 656.167561] ? __schedule+0x329/0x8a0 [ 656.170443] ? sched_clock_cpu+0xc/0xc0 [ 656.173423] schedule+0x3a/0xb0 [ 656.175951] schedule_timeout+0x1b8/0x3c0 [ 656.179015] ? __next_timer_interrupt+0xd0/0xd0 [ 656.182388] rcu_gp_kthread+0x4a8/0xa60 [ 656.185315] kthread+0x108/0x140 [ 656.187846] ? rcu_barrier_func+0xa0/0xa0 [ 656.190867] ? kthread_park+0x80/0x80 [ 656.193670] ret_from_fork+0x3a/0x50 [ 667.226908] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [stress:1826] [ 667.231966] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 667.268780] irq event stamp: 132973063 [ 667.271799] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 667.278831] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 667.285577] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 667.291892] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 667.297918] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 667.303559] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 667.309724] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 667.313921] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 667.327436] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 667.333039] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 667.338363] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 667.343690] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 667.349042] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 667.349906] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 667.354391] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 667.356718] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 667.362095] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 667.378075] irq event stamp: 136226147 [ 667.378079] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 667.384125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 667.385765] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 667.385768] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 667.392435] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 667.392438] Call Trace: [ 667.394554] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 667.394556] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 667.401205] do_raw_spin_lock+0xab/0xb0 [ 667.404017] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 667.404020] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 667.409468] _raw_spin_lock+0x63/0x80 [ 667.410825] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 667.417066] z3fold_zpool_shrink+0x2d3/0x570 [ 667.419652] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 667.422983] ? sched_clock_cpu+0xc/0xc0 [ 667.425759] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 667.425760] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 667.430188] zswap_frontswap_store+0x424/0x7c1 [ 667.431863] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 667.431864] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 667.445843] __frontswap_store+0xc4/0x162 [ 667.447706] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 667.447709] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 667.453526] swap_writepage+0x39/0x70 [ 667.455309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 667.455310] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 667.460868] pageout.isra.0+0x12c/0x5d0 [ 667.463386] Call Trace: [ 667.467159] shrink_page_list+0x1124/0x1830 [ 667.469658] do_raw_spin_lock+0xab/0xb0 [ 667.475142] shrink_inactive_list+0x1da/0x460 [ 667.476902] _raw_spin_lock+0x63/0x80 [ 667.482382] shrink_node_memcg+0x202/0x770 [ 667.485062] __swp_swapcount+0xb9/0xf0 [ 667.488287] ? sched_clock_cpu+0xc/0xc0 [ 667.490440] __read_swap_cache_async+0xc0/0x3e0 [ 667.495924] shrink_node+0xdf/0x490 [ 667.497645] swap_cluster_readahead+0x184/0x330 [ 667.500075] do_try_to_free_pages+0xdb/0x3c0 [ 667.501848] ? find_held_lock+0x32/0x90 [ 667.505144] try_to_free_pages+0x112/0x2e0 [ 667.506927] swapin_readahead+0x2b4/0x4e0 [ 667.510090] __alloc_pages_slowpath+0x422/0x1000 [ 667.511796] ? sched_clock_cpu+0xc/0xc0 [ 667.515005] ? __lock_acquire+0x247/0x1900 [ 667.516642] do_swap_page+0x3ac/0xc20 [ 667.520350] __alloc_pages_nodemask+0x37f/0x400 [ 667.521894] __handle_mm_fault+0x8da/0x1900 [ 667.525591] alloc_pages_vma+0xcc/0x170 [ 667.527298] handle_mm_fault+0x159/0x340 [ 667.530493] __read_swap_cache_async+0x1e9/0x3e0 [ 667.532124] do_user_addr_fault+0x1fe/0x480 [ 667.535393] swap_cluster_readahead+0x184/0x330 [ 667.537107] do_page_fault+0x31/0x210 [ 667.540265] ? find_held_lock+0x32/0x90 [ 667.541832] page_fault+0x3e/0x50 [ 667.544840] swapin_readahead+0x2b4/0x4e0 [ 667.546491] RIP: 0033:0x563ef1681298 [ 667.549812] ? sched_clock_cpu+0xc/0xc0 [ 667.551285] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 667.554417] do_swap_page+0x3ac/0xc20 [ 667.556047] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 667.559335] __handle_mm_fault+0x8da/0x1900 [ 667.560923] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 667.560924] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 667.563856] handle_mm_fault+0x159/0x340 [ 667.565275] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 667.565276] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 667.567981] do_user_addr_fault+0x1fe/0x480 [ 667.569422] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 667.630452] do_page_fault+0x31/0x210 [ 667.633334] page_fault+0x3e/0x50 [ 667.635987] RIP: 0033:0x563ef1681298 [ 667.638576] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 667.651292] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 667.655200] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 667.660343] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 667.665494] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 667.670644] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 667.675801] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 683.288871] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [stress:1823] [ 683.291988] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 683.314090] irq event stamp: 156337142 [ 683.316084] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 683.320148] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 683.323844] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 683.327642] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 683.331284] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 683.334714] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 683.338437] RIP: 0010:queued_spin_lock_slowpath+0x126/0x1e0 [ 683.341122] Code: 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 8b 07 <66> 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 48 85 [ 683.349232] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 683.352659] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 683.355945] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 683.359241] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 683.362533] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 683.365829] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 683.369133] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 683.372773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 683.375592] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 683.378900] Call Trace: [ 683.380537] do_raw_spin_lock+0xab/0xb0 [ 683.382656] _raw_spin_lock+0x63/0x80 [ 683.384708] __swap_duplicate+0x163/0x220 [ 683.386885] __read_swap_cache_async+0x1f9/0x3e0 [ 683.389270] swap_cluster_readahead+0x184/0x330 [ 683.391622] ? find_held_lock+0x32/0x90 [ 683.393720] swapin_readahead+0x2b4/0x4e0 [ 683.395872] ? sched_clock_cpu+0xc/0xc0 [ 683.397966] do_swap_page+0x3ac/0xc20 [ 683.399987] __handle_mm_fault+0x8da/0x1900 [ 683.402189] handle_mm_fault+0x159/0x340 [ 683.404289] do_user_addr_fault+0x1fe/0x480 [ 683.406484] do_page_fault+0x31/0x210 [ 683.408490] page_fault+0x3e/0x50 [ 683.410358] RIP: 0033:0x563ef1681298 [ 683.412299] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 683.420312] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 683.422863] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 683.426099] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 683.429330] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 683.432565] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 683.435792] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 695.226844] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [stress:1826] [ 695.232063] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 695.269755] irq event stamp: 132973063 [ 695.272836] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 695.279877] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 695.286632] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 695.292945] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 695.298981] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 695.304623] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 695.310750] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 695.314930] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 695.328301] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 695.333915] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 695.339196] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 695.344509] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 695.349831] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 695.349842] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 695.355160] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 695.355163] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 695.357452] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 695.362790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 695.365329] irq event stamp: 136226147 [ 695.365333] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 695.403912] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 695.406023] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 695.406025] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 695.408733] Call Trace: [ 695.411604] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 695.411606] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 695.416011] do_raw_spin_lock+0xab/0xb0 [ 695.418875] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 695.418878] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 695.424243] _raw_spin_lock+0x63/0x80 [ 695.425650] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 695.430704] z3fold_zpool_shrink+0x2d3/0x570 [ 695.433344] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 695.436095] ? sched_clock_cpu+0xc/0xc0 [ 695.438908] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 695.438909] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 695.442559] zswap_frontswap_store+0x424/0x7c1 [ 695.444254] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 695.444255] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 695.455482] __frontswap_store+0xc4/0x162 [ 695.457351] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 695.457354] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 695.462050] swap_writepage+0x39/0x70 [ 695.463827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 695.463828] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 695.468342] pageout.isra.0+0x12c/0x5d0 [ 695.470854] Call Trace: [ 695.473965] shrink_page_list+0x1124/0x1830 [ 695.476456] do_raw_spin_lock+0xab/0xb0 [ 695.480948] shrink_inactive_list+0x1da/0x460 [ 695.482704] _raw_spin_lock+0x63/0x80 [ 695.487174] shrink_node_memcg+0x202/0x770 [ 695.489843] __swp_swapcount+0xb9/0xf0 [ 695.492502] ? sched_clock_cpu+0xc/0xc0 [ 695.494646] __read_swap_cache_async+0xc0/0x3e0 [ 695.499003] shrink_node+0xdf/0x490 [ 695.500713] swap_cluster_readahead+0x184/0x330 [ 695.502741] do_try_to_free_pages+0xdb/0x3c0 [ 695.504517] ? find_held_lock+0x32/0x90 [ 695.507247] try_to_free_pages+0x112/0x2e0 [ 695.509027] swapin_readahead+0x2b4/0x4e0 [ 695.511616] __alloc_pages_slowpath+0x422/0x1000 [ 695.513311] ? sched_clock_cpu+0xc/0xc0 [ 695.515942] ? __lock_acquire+0x247/0x1900 [ 695.517576] do_swap_page+0x3ac/0xc20 [ 695.520559] __alloc_pages_nodemask+0x37f/0x400 [ 695.522101] __handle_mm_fault+0x8da/0x1900 [ 695.525098] alloc_pages_vma+0xcc/0x170 [ 695.526805] handle_mm_fault+0x159/0x340 [ 695.529426] __read_swap_cache_async+0x1e9/0x3e0 [ 695.531061] do_user_addr_fault+0x1fe/0x480 [ 695.533727] swap_cluster_readahead+0x184/0x330 [ 695.535448] do_page_fault+0x31/0x210 [ 695.538580] ? find_held_lock+0x32/0x90 [ 695.540147] page_fault+0x3e/0x50 [ 695.543155] swapin_readahead+0x2b4/0x4e0 [ 695.544799] RIP: 0033:0x563ef1681298 [ 695.548123] ? sched_clock_cpu+0xc/0xc0 [ 695.549589] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 695.552738] do_swap_page+0x3ac/0xc20 [ 695.554366] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 695.557657] __handle_mm_fault+0x8da/0x1900 [ 695.559255] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 695.559256] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 695.562202] handle_mm_fault+0x159/0x340 [ 695.563613] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 695.563614] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 695.566303] do_user_addr_fault+0x1fe/0x480 [ 695.567743] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 695.628681] do_page_fault+0x31/0x210 [ 695.631561] page_fault+0x3e/0x50 [ 695.634220] RIP: 0033:0x563ef1681298 [ 695.637023] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 695.649337] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 695.653219] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 695.658306] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 695.663390] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 695.668472] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 695.673563] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 711.288807] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [stress:1823] [ 711.293868] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 711.330956] irq event stamp: 156337142 [ 711.334026] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 711.340806] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 711.346919] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 711.353223] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 711.359256] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 711.364928] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 711.371090] RIP: 0010:queued_spin_lock_slowpath+0x124/0x1e0 [ 711.375096] Code: 00 89 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 [ 711.388552] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 711.394145] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 711.399471] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 711.404793] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 711.410132] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 711.415469] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 711.420814] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 711.426760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 711.431179] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 711.436526] Call Trace: [ 711.438865] do_raw_spin_lock+0xab/0xb0 [ 711.442095] _raw_spin_lock+0x63/0x80 [ 711.445194] __swap_duplicate+0x163/0x220 [ 711.448508] __read_swap_cache_async+0x1f9/0x3e0 [ 711.452216] swap_cluster_readahead+0x184/0x330 [ 711.455862] ? find_held_lock+0x32/0x90 [ 711.459066] swapin_readahead+0x2b4/0x4e0 [ 711.462369] ? sched_clock_cpu+0xc/0xc0 [ 711.465559] do_swap_page+0x3ac/0xc20 [ 711.468624] __handle_mm_fault+0x8da/0x1900 [ 711.472017] handle_mm_fault+0x159/0x340 [ 711.475227] do_user_addr_fault+0x1fe/0x480 [ 711.478606] do_page_fault+0x31/0x210 [ 711.481417] page_fault+0x3e/0x50 [ 711.484221] RIP: 0033:0x563ef1681298 [ 711.487169] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 711.500400] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 711.504431] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 711.509704] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 711.514957] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 711.520210] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 711.525364] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 723.226780] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [stress:1826] [ 723.231985] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 723.269737] irq event stamp: 132973063 [ 723.272869] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 723.280017] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 723.286778] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 723.293170] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 723.299154] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 723.304852] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 723.311075] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 723.315305] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 723.328629] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 723.334225] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 723.339539] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 723.344866] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 723.349779] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 723.350202] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 723.350203] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 723.352537] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 723.357772] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 723.360117] irq event stamp: 136226147 [ 723.360121] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 723.398439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 723.401113] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 723.401115] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 723.404343] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 723.407254] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 723.407256] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 723.411803] Call Trace: [ 723.414711] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 723.414714] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 723.421063] do_raw_spin_lock+0xab/0xb0 [ 723.423561] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 723.429829] _raw_spin_lock+0x63/0x80 [ 723.432495] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 723.434977] z3fold_zpool_shrink+0x2d3/0x570 [ 723.437831] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 723.437832] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 723.442279] ? sched_clock_cpu+0xc/0xc0 [ 723.444032] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 723.444033] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 723.457792] zswap_frontswap_store+0x424/0x7c1 [ 723.459530] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 723.459533] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 723.465336] __frontswap_store+0xc4/0x162 [ 723.467229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 723.467230] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 723.472777] swap_writepage+0x39/0x70 [ 723.475335] Call Trace: [ 723.478622] pageout.isra.0+0x12c/0x5d0 [ 723.481168] do_raw_spin_lock+0xab/0xb0 [ 723.486677] shrink_page_list+0x1124/0x1830 [ 723.488577] _raw_spin_lock+0x63/0x80 [ 723.494071] shrink_inactive_list+0x1da/0x460 [ 723.496787] __swp_swapcount+0xb9/0xf0 [ 723.500232] shrink_node_memcg+0x202/0x770 [ 723.502404] __read_swap_cache_async+0xc0/0x3e0 [ 723.507771] ? sched_clock_cpu+0xc/0xc0 [ 723.509469] swap_cluster_readahead+0x184/0x330 [ 723.511898] shrink_node+0xdf/0x490 [ 723.513612] ? find_held_lock+0x32/0x90 [ 723.516919] do_try_to_free_pages+0xdb/0x3c0 [ 723.518684] swapin_readahead+0x2b4/0x4e0 [ 723.521741] try_to_free_pages+0x112/0x2e0 [ 723.523524] ? sched_clock_cpu+0xc/0xc0 [ 723.526719] __alloc_pages_slowpath+0x422/0x1000 [ 723.528430] do_swap_page+0x3ac/0xc20 [ 723.532013] ? __lock_acquire+0x247/0x1900 [ 723.533652] __handle_mm_fault+0x8da/0x1900 [ 723.537347] __alloc_pages_nodemask+0x37f/0x400 [ 723.538895] handle_mm_fault+0x159/0x340 [ 723.541982] alloc_pages_vma+0xcc/0x170 [ 723.543673] do_user_addr_fault+0x1fe/0x480 [ 723.546977] __read_swap_cache_async+0x1e9/0x3e0 [ 723.548590] do_page_fault+0x31/0x210 [ 723.551729] swap_cluster_readahead+0x184/0x330 [ 723.553440] page_fault+0x3e/0x50 [ 723.556448] ? find_held_lock+0x32/0x90 [ 723.558019] RIP: 0033:0x563ef1681298 [ 723.561336] swapin_readahead+0x2b4/0x4e0 [ 723.562981] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 723.566116] ? sched_clock_cpu+0xc/0xc0 [ 723.567579] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 723.570859] do_swap_page+0x3ac/0xc20 [ 723.572484] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 723.572485] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 723.575422] __handle_mm_fault+0x8da/0x1900 [ 723.577016] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 723.577017] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 723.579665] handle_mm_fault+0x159/0x340 [ 723.581079] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 723.640985] do_user_addr_fault+0x1fe/0x480 [ 723.644160] do_page_fault+0x31/0x210 [ 723.647004] page_fault+0x3e/0x50 [ 723.649602] RIP: 0033:0x563ef1681298 [ 723.652377] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 723.665363] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 723.669229] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 723.674314] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 723.679404] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 723.684500] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 723.689588] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 739.288743] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [stress:1823] [ 739.291738] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 739.313003] irq event stamp: 156337142 [ 739.314950] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 739.318881] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 739.322444] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 739.326109] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 739.329622] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 739.332948] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 739.336551] RIP: 0010:queued_spin_lock_slowpath+0x124/0x1e0 [ 739.339158] Code: 00 89 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 [ 739.346985] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 739.350315] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 739.353499] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 739.356688] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 739.359872] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 739.363065] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 739.366273] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 739.369807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 739.372530] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 739.375718] Call Trace: [ 739.377307] do_raw_spin_lock+0xab/0xb0 [ 739.379367] _raw_spin_lock+0x63/0x80 [ 739.381369] __swap_duplicate+0x163/0x220 [ 739.383484] __read_swap_cache_async+0x1f9/0x3e0 [ 739.385795] swap_cluster_readahead+0x184/0x330 [ 739.388074] ? find_held_lock+0x32/0x90 [ 739.390108] swapin_readahead+0x2b4/0x4e0 [ 739.392201] ? sched_clock_cpu+0xc/0xc0 [ 739.394224] do_swap_page+0x3ac/0xc20 [ 739.396182] __handle_mm_fault+0x8da/0x1900 [ 739.398310] handle_mm_fault+0x159/0x340 [ 739.400350] do_user_addr_fault+0x1fe/0x480 [ 739.402479] do_page_fault+0x31/0x210 [ 739.404431] page_fault+0x3e/0x50 [ 739.406248] RIP: 0033:0x563ef1681298 [ 739.408135] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 739.415852] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 739.418326] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 739.421450] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 739.424576] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 739.427691] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 739.430806] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 751.226717] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [stress:1826] [ 751.231956] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 751.269802] irq event stamp: 132973063 [ 751.272906] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 751.279977] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 751.286758] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 751.293098] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 751.299149] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 751.304803] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 751.310958] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 751.315155] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 751.328594] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 751.334163] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 751.339448] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 751.344737] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 751.349715] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 751.350025] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 751.352332] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 751.357599] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 751.357602] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 751.373458] irq event stamp: 136226147 [ 751.373461] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 751.378820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 751.378821] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 751.381459] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 751.381461] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 751.384671] Call Trace: [ 751.387545] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 751.387547] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 751.392066] do_raw_spin_lock+0xab/0xb0 [ 751.394519] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 751.394522] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 751.401121] _raw_spin_lock+0x63/0x80 [ 751.403960] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 751.406400] z3fold_zpool_shrink+0x2d3/0x570 [ 751.409164] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 751.414991] ? sched_clock_cpu+0xc/0xc0 [ 751.416745] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 751.416746] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 751.423085] zswap_frontswap_store+0x424/0x7c1 [ 751.425202] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 751.425203] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 751.428419] __frontswap_store+0xc4/0x162 [ 751.434430] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 751.434434] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 751.438091] swap_writepage+0x39/0x70 [ 751.440722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 751.440724] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 751.444111] pageout.isra.0+0x12c/0x5d0 [ 751.446645] Call Trace: [ 751.452128] shrink_page_list+0x1124/0x1830 [ 751.454023] do_raw_spin_lock+0xab/0xb0 [ 751.459485] shrink_inactive_list+0x1da/0x460 [ 751.461962] _raw_spin_lock+0x63/0x80 [ 751.465386] shrink_node_memcg+0x202/0x770 [ 751.467845] __swp_swapcount+0xb9/0xf0 [ 751.473877] ? sched_clock_cpu+0xc/0xc0 [ 751.475561] __read_swap_cache_async+0xc0/0x3e0 [ 751.480101] shrink_node+0xdf/0x490 [ 751.482575] swap_cluster_readahead+0x184/0x330 [ 751.485879] do_try_to_free_pages+0xdb/0x3c0 [ 751.487273] ? find_held_lock+0x32/0x90 [ 751.490776] try_to_free_pages+0x112/0x2e0 [ 751.492456] swapin_readahead+0x2b4/0x4e0 [ 751.496057] __alloc_pages_slowpath+0x422/0x1000 [ 751.497670] ? sched_clock_cpu+0xc/0xc0 [ 751.501074] ? __lock_acquire+0x247/0x1900 [ 751.502696] do_swap_page+0x3ac/0xc20 [ 751.504137] __alloc_pages_nodemask+0x37f/0x400 [ 751.507849] __handle_mm_fault+0x8da/0x1900 [ 751.509433] alloc_pages_vma+0xcc/0x170 [ 751.513081] handle_mm_fault+0x159/0x340 [ 751.514805] __read_swap_cache_async+0x1e9/0x3e0 [ 751.517971] do_user_addr_fault+0x1fe/0x480 [ 751.519615] swap_cluster_readahead+0x184/0x330 [ 751.522842] do_page_fault+0x31/0x210 [ 751.524576] ? find_held_lock+0x32/0x90 [ 751.527666] page_fault+0x3e/0x50 [ 751.529264] swapin_readahead+0x2b4/0x4e0 [ 751.532225] RIP: 0033:0x563ef1681298 [ 751.533899] ? sched_clock_cpu+0xc/0xc0 [ 751.537186] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 751.538680] do_swap_page+0x3ac/0xc20 [ 751.541781] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 751.543450] __handle_mm_fault+0x8da/0x1900 [ 751.546693] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 751.546694] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 751.548336] handle_mm_fault+0x159/0x340 [ 751.551235] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 751.552680] do_user_addr_fault+0x1fe/0x480 [ 751.555331] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 751.555332] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 751.556807] do_page_fault+0x31/0x210 [ 751.586183] page_fault+0x3e/0x50 [ 751.587456] RIP: 0033:0x563ef1681298 [ 751.588782] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 751.594193] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 751.595958] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 751.598167] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 751.600378] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 751.602599] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 751.604814] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 767.288680] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [stress:1823] [ 767.293741] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 767.330790] irq event stamp: 156337142 [ 767.333871] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 767.340593] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 767.346722] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 767.353023] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 767.359040] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 767.364691] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 767.370742] RIP: 0010:queued_spin_lock_slowpath+0x124/0x1e0 [ 767.375027] Code: 00 89 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 [ 767.388456] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 767.393959] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 767.399283] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 767.404514] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 767.409842] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 767.415104] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 767.420389] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 767.426255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 767.430721] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 767.436085] Call Trace: [ 767.438427] do_raw_spin_lock+0xab/0xb0 [ 767.441650] _raw_spin_lock+0x63/0x80 [ 767.444760] __swap_duplicate+0x163/0x220 [ 767.448103] __read_swap_cache_async+0x1f9/0x3e0 [ 767.451805] swap_cluster_readahead+0x184/0x330 [ 767.455464] ? find_held_lock+0x32/0x90 [ 767.458666] swapin_readahead+0x2b4/0x4e0 [ 767.461875] ? sched_clock_cpu+0xc/0xc0 [ 767.465058] do_swap_page+0x3ac/0xc20 [ 767.468041] __handle_mm_fault+0x8da/0x1900 [ 767.471437] handle_mm_fault+0x159/0x340 [ 767.474666] do_user_addr_fault+0x1fe/0x480 [ 767.478065] do_page_fault+0x31/0x210 [ 767.481030] page_fault+0x3e/0x50 [ 767.483844] RIP: 0033:0x563ef1681298 [ 767.486808] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 767.500032] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 767.504079] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 767.509357] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 767.514671] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 767.519835] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 767.525012] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 779.226654] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [stress:1826] [ 779.231853] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 779.269418] irq event stamp: 132973063 [ 779.272498] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 779.279512] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 779.286246] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 779.292530] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 779.298564] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 779.304235] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 779.310394] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 779.314594] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 779.328019] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 779.333577] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 779.338864] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 779.344149] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 779.349438] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 779.349655] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 779.354721] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 779.357339] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 779.362609] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 779.380908] irq event stamp: 136226147 [ 779.380911] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 779.386906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 779.388716] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 779.388719] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 779.395353] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 779.395356] Call Trace: [ 779.397737] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 779.397740] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 779.404331] do_raw_spin_lock+0xab/0xb0 [ 779.407548] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 779.407552] RIP: 0010:queued_spin_lock_slowpath+0x44/0x1e0 [ 779.412991] _raw_spin_lock+0x63/0x80 [ 779.414520] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 c3 f6 [ 779.420745] z3fold_zpool_shrink+0x2d3/0x570 [ 779.423699] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 779.427056] ? sched_clock_cpu+0xc/0xc0 [ 779.430231] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 779.430232] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 779.434635] zswap_frontswap_store+0x424/0x7c1 [ 779.436483] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 779.436484] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 779.450318] __frontswap_store+0xc4/0x162 [ 779.452368] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 779.452371] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 779.458146] swap_writepage+0x39/0x70 [ 779.460098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 779.460099] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 779.465598] pageout.isra.0+0x12c/0x5d0 [ 779.468423] Call Trace: [ 779.472166] shrink_page_list+0x1124/0x1830 [ 779.474988] do_raw_spin_lock+0xab/0xb0 [ 779.480435] shrink_inactive_list+0x1da/0x460 [ 779.482373] _raw_spin_lock+0x63/0x80 [ 779.487808] shrink_node_memcg+0x202/0x770 [ 779.490840] __swp_swapcount+0xb9/0xf0 [ 779.494033] ? sched_clock_cpu+0xc/0xc0 [ 779.496442] __read_swap_cache_async+0xc0/0x3e0 [ 779.501877] shrink_node+0xdf/0x490 [ 779.503764] swap_cluster_readahead+0x184/0x330 [ 779.506180] do_try_to_free_pages+0xdb/0x3c0 [ 779.508146] ? find_held_lock+0x32/0x90 [ 779.511423] try_to_free_pages+0x112/0x2e0 [ 779.513401] swapin_readahead+0x2b4/0x4e0 [ 779.516537] __alloc_pages_slowpath+0x422/0x1000 [ 779.518430] ? sched_clock_cpu+0xc/0xc0 [ 779.521604] ? __lock_acquire+0x247/0x1900 [ 779.523408] do_swap_page+0x3ac/0xc20 [ 779.527086] __alloc_pages_nodemask+0x37f/0x400 [ 779.528781] __handle_mm_fault+0x8da/0x1900 [ 779.532437] alloc_pages_vma+0xcc/0x170 [ 779.534340] handle_mm_fault+0x159/0x340 [ 779.536788] __read_swap_cache_async+0x1e9/0x3e0 [ 779.536790] swap_cluster_readahead+0x184/0x330 [ 779.536792] ? find_held_lock+0x32/0x90 [ 779.536795] swapin_readahead+0x2b4/0x4e0 [ 779.536797] ? sched_clock_cpu+0xc/0xc0 [ 779.536800] do_swap_page+0x3ac/0xc20 [ 779.536803] __handle_mm_fault+0x8da/0x1900 [ 779.536807] handle_mm_fault+0x159/0x340 [ 779.536809] do_user_addr_fault+0x1fe/0x480 [ 779.536812] do_page_fault+0x31/0x210 [ 779.536817] page_fault+0x3e/0x50 [ 779.536819] RIP: 0033:0x563ef1681298 [ 779.536821] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 779.536822] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 779.536823] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 779.536823] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 779.536824] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 779.536824] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 779.536825] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 779.646979] do_user_addr_fault+0x1fe/0x480 [ 779.650091] do_page_fault+0x31/0x210 [ 779.652864] page_fault+0x3e/0x50 [ 779.655417] RIP: 0033:0x563ef1681298 [ 779.658128] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 779.670957] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 779.674752] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 779.679760] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 779.684772] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 779.689790] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 779.694808] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 795.288617] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [stress:1823] [ 795.293637] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 795.330349] irq event stamp: 156337142 [ 795.333336] hardirqs last enabled at (156337141): [<ffffffff90001c6a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 795.340013] hardirqs last disabled at (156337142): [<ffffffff909cef64>] __schedule+0xc4/0x8a0 [ 795.346026] softirqs last enabled at (156335334): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 795.352222] softirqs last disabled at (156335327): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 795.358126] CPU: 1 PID: 1823 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 795.363661] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 795.369697] RIP: 0010:queued_spin_lock_slowpath+0x124/0x1e0 [ 795.373878] Code: 00 89 1d 00 eb a1 41 83 c0 01 c1 e1 10 41 c1 e0 12 44 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 64 c6 07 01 [ 795.387088] RSP: 0000:ffffa6f1c0907b90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 795.392576] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 0000000000080000 [ 795.397795] RDX: ffff8d047e7ec400 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 795.403018] RBP: ffff8d047539ad40 R08: 0000000000080000 R09: 0000000000000000 [ 795.408239] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d047539ad58 [ 795.413472] R13: ffffa6f1c0907c7f R14: 0000000000000040 R15: fffff5cac058e800 [ 795.418705] FS: 00007f35ba5bb740(0000) GS:ffff8d047e600000(0000) knlGS:0000000000000000 [ 795.424553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 795.428929] CR2: 0000000000000008 CR3: 0000000031906004 CR4: 0000000000160ee0 [ 795.434189] Call Trace: [ 795.436497] do_raw_spin_lock+0xab/0xb0 [ 795.439681] _raw_spin_lock+0x63/0x80 [ 795.442751] __swap_duplicate+0x163/0x220 [ 795.446022] __read_swap_cache_async+0x1f9/0x3e0 [ 795.449662] swap_cluster_readahead+0x184/0x330 [ 795.453244] ? find_held_lock+0x32/0x90 [ 795.456386] swapin_readahead+0x2b4/0x4e0 [ 795.459635] ? sched_clock_cpu+0xc/0xc0 [ 795.462772] do_swap_page+0x3ac/0xc20 [ 795.465797] __handle_mm_fault+0x8da/0x1900 [ 795.469149] handle_mm_fault+0x159/0x340 [ 795.472335] do_user_addr_fault+0x1fe/0x480 [ 795.475682] do_page_fault+0x31/0x210 [ 795.478700] page_fault+0x3e/0x50 [ 795.481482] RIP: 0033:0x563ef1681298 [ 795.484411] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 795.497620] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 795.501612] RAX: 00000000012eb000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 795.506845] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 795.512045] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 795.517248] R10: 00007f35b0036010 R11: 0000000000000246 R12: 0000563ef1683004 [ 795.522439] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 807.226589] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [stress:1826] [ 807.232011] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 807.269765] irq event stamp: 132973063 [ 807.272859] hardirqs last enabled at (132973063): [<ffffffff909d6b63>] _raw_spin_unlock_irqrestore+0x43/0x50 [ 807.279892] hardirqs last disabled at (132973062): [<ffffffff909d6916>] _raw_spin_lock_irqsave+0x16/0x80 [ 807.286620] softirqs last enabled at (132972610): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 807.292926] softirqs last disabled at (132972603): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 807.298951] CPU: 0 PID: 1826 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 807.304573] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 807.310681] RIP: 0010:queued_spin_lock_slowpath+0x42/0x1e0 [ 807.314865] Code: 49 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 [ 807.328186] RSP: 0000:ffffa6f1c094f478 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 807.333715] RAX: 0000000000000101 RBX: ffff8d046b5f7010 RCX: 8888888888888889 [ 807.338971] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d046b5f7010 [ 807.344216] RBP: ffff8d046b5f7010 R08: 000000899932efc2 R09: 0000000000000000 [ 807.349466] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d046b5f7028 [ 807.349588] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress:1824] [ 807.354711] R13: fffff5cac0ad7dc8 R14: ffff8d0475f5ba08 R15: ffff8d0475f5baa0 [ 807.357021] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon virtio_net net_failover intel_agp failover intel_gtt qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw virtio_console virtio_blk agpgart qemu_fw_cfg [ 807.362284] FS: 00007f35ba5bb740(0000) GS:ffff8d047e400000(0000) knlGS:0000000000000000 [ 807.378179] irq event stamp: 136226147 [ 807.378183] hardirqs last enabled at (136226147): [<ffffffff902b81d4>] handle_mm_fault+0x1b4/0x340 [ 807.384157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 807.385805] hardirqs last disabled at (136226146): [<ffffffff902b80cd>] handle_mm_fault+0xad/0x340 [ 807.385807] softirqs last enabled at (136225360): [<ffffffff90c00351>] __do_softirq+0x351/0x451 [ 807.392390] CR2: 00007f35af1b5010 CR3: 0000000004d52003 CR4: 0000000000160ef0 [ 807.392393] Call Trace: [ 807.394519] softirqs last disabled at (136225349): [<ffffffff900c9821>] irq_exit+0xf1/0x100 [ 807.394521] CPU: 2 PID: 1824 Comm: stress Tainted: G D W L 5.3.0-rc5+ #71 [ 807.401076] do_raw_spin_lock+0xab/0xb0 [ 807.403913] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014 [ 807.403915] RIP: 0010:queued_spin_lock_slowpath+0x44/0x1e0 [ 807.409332] _raw_spin_lock+0x63/0x80 [ 807.410736] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 23 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 65 48 ff 05 e8 f7 09 70 c3 f6 [ 807.416931] z3fold_zpool_shrink+0x2d3/0x570 [ 807.419557] RSP: 0000:ffffa6f1c0927ba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 807.422904] ? sched_clock_cpu+0xc/0xc0 [ 807.425705] RAX: 0000000000080101 RBX: ffff8d047539ad40 RCX: 8888888888888889 [ 807.425707] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d047539ad40 [ 807.430085] zswap_frontswap_store+0x424/0x7c1 [ 807.431769] RBP: ffff8d047539ad40 R08: 000000899932bdf5 R09: 0000000000000000 [ 807.431770] R10: 0000000000000002 R11: 0000000000000000 R12: ffff8d047539ad58 [ 807.445456] __frontswap_store+0xc4/0x162 [ 807.447327] R13: ffff8d047539ad40 R14: 0000000000002120 R15: 0000000000000000 [ 807.447330] FS: 00007f35ba5bb740(0000) GS:ffff8d047e800000(0000) knlGS:0000000000000000 [ 807.453070] swap_writepage+0x39/0x70 [ 807.454846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 807.454847] CR2: 00007f35af030010 CR3: 00000000389e4001 CR4: 0000000000160ee0 [ 807.460327] pageout.isra.0+0x12c/0x5d0 [ 807.462841] Call Trace: [ 807.466574] shrink_page_list+0x1124/0x1830 [ 807.469067] do_raw_spin_lock+0xab/0xb0 [ 807.474474] shrink_inactive_list+0x1da/0x460 [ 807.476233] _raw_spin_lock+0x63/0x80 [ 807.481636] shrink_node_memcg+0x202/0x770 [ 807.484319] __swp_swapcount+0xb9/0xf0 [ 807.487501] ? sched_clock_cpu+0xc/0xc0 [ 807.489647] __read_swap_cache_async+0xc0/0x3e0 [ 807.495050] shrink_node+0xdf/0x490 [ 807.496761] swap_cluster_readahead+0x184/0x330 [ 807.499157] do_try_to_free_pages+0xdb/0x3c0 [ 807.500928] ? find_held_lock+0x32/0x90 [ 807.504180] try_to_free_pages+0x112/0x2e0 [ 807.505961] swapin_readahead+0x2b4/0x4e0 [ 807.509078] __alloc_pages_slowpath+0x422/0x1000 [ 807.510779] ? sched_clock_cpu+0xc/0xc0 [ 807.513939] ? __lock_acquire+0x247/0x1900 [ 807.515575] do_swap_page+0x3ac/0xc20 [ 807.518005] __alloc_pages_nodemask+0x37f/0x400 [ 807.519561] __handle_mm_fault+0x8da/0x1900 [ 807.521417] alloc_pages_vma+0xcc/0x170 [ 807.523135] handle_mm_fault+0x159/0x340 [ 807.524774] __read_swap_cache_async+0x1e9/0x3e0 [ 807.526415] do_user_addr_fault+0x1fe/0x480 [ 807.528062] swap_cluster_readahead+0x184/0x330 [ 807.529785] do_page_fault+0x31/0x210 [ 807.531359] ? find_held_lock+0x32/0x90 [ 807.532940] page_fault+0x3e/0x50 [ 807.534450] swapin_readahead+0x2b4/0x4e0 [ 807.536106] RIP: 0033:0x563ef1681298 [ 807.537722] ? sched_clock_cpu+0xc/0xc0 [ 807.539207] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 807.540741] do_swap_page+0x3ac/0xc20 [ 807.542385] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 807.543977] __handle_mm_fault+0x8da/0x1900 [ 807.545579] RAX: 00000000002e4000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 807.545580] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 807.547034] handle_mm_fault+0x159/0x340 [ 807.548456] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 807.548457] R10: 00007f35af02f010 R11: 0000000000000246 R12: 0000563ef1683004 [ 807.549794] do_user_addr_fault+0x1fe/0x480 [ 807.551247] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800 [ 807.578558] do_page_fault+0x31/0x210 [ 807.579966] page_fault+0x3e/0x50 [ 807.581273] RIP: 0033:0x563ef1681298 [ 807.582635] Code: 7e 01 00 00 89 df e8 47 e1 ff ff 44 8b 2d 84 4d 00 00 4d 85 ff 7e 40 31 c0 eb 0f 0f 1f 80 00 00 00 00 4c 01 f0 49 39 c7 7e 2d <80> 7c 05 00 5a 4c 8d 54 05 00 74 ec 4c 89 14 24 45 85 ed 0f 89 de [ 807.588202] RSP: 002b:00007ffcd50ca7b0 EFLAGS: 00010206 [ 807.590022] RAX: 0000000008aff000 RBX: ffffffffffffffff RCX: 00007f35ba6b4156 [ 807.592300] RDX: 0000000000000000 RSI: 000000000b86f000 RDI: 0000000000000000 [ 807.594575] RBP: 00007f35aed4c010 R08: 00007f35aed4c010 R09: 0000000000000000 [ 807.596857] R10: 00007f35b784a010 R11: 0000000000000246 R12: 0000563ef1683004 [ 807.599133] R13: 0000000000000002 R14: 0000000000001000 R15: 000000000b86e800
