Hi, I’m randomly encountering a persistent soft lockup issue in the NFS client on Kernel version 6.11.2. The issue appeared for the first time on the Kernel version 6.11, not happening before. The problem appears during NFS writeback operations, resulting in a system lockup. Below are the details: Environment Details: Kernel: 6.11.2 NFS Version: v4.2, NFS server: NetAPP AFF A800 System: CentOS Stream 9, KVM virtualization Problem Description: The system experiences soft lockups during NFS operations, specifically when attempting to flush dirty pages to the server. The issue appears to stem from NFS locking mechanisms not being properly released, causing processes to stall indefinitely. This results in the affected core's 100% system CPU utilization with no idle time. Behavior: The system CPU remains fully occupied (100% system usage) due to stuck NFS locks, with the following stack trace repeated in the logs: ``` RIP: 0010:nfs_unlock_request+0x12/0x30 [nfs] nfs_lock_and_join_requests+0x67/0x240 [nfs] nfs_page_async_flush+0x1b/0x210 [nfs] ... write_cache_pages+0x57/0xb0 nfs_writepages+0x117/0x2b0 [nfs] ``` Logs: See attached log file including the complete logs timeline ending with the system shutdown (reboot). Can you help me to identify the issue and find the fix for the described situation? I’m here to assist you in getting more information in our environment. Thank you for your time and assistance. Best regards, Michal Slabihoudek michal.slabihoudek@xxxxxxxxxxxx
2024-10-19 09:03:34.394 [855955.206792] </TASK> 2024-10-19 09:03:34.394 [855955.206480] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:03:34.394 [855955.206159] ? __pfx_kthread+0x10/0x10 2024-10-19 09:03:34.394 [855955.205852] ret_from_fork+0x2d/0x50 2024-10-19 09:03:34.394 [855955.205538] ? __pfx_kthread+0x10/0x10 2024-10-19 09:03:34.394 [855955.205238] kthread+0xcc/0x100 2024-10-19 09:03:34.394 [855955.204857] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:03:34.394 [855955.204531] worker_thread+0x239/0x340 2024-10-19 09:03:34.393 [855955.204117] process_one_work+0x179/0x390 2024-10-19 09:03:34.393 [855955.203795] wb_workfn+0x45/0x150 2024-10-19 09:03:34.393 [855955.203445] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:03:34.393 [855955.203089] wb_writeback+0x180/0x2f0 2024-10-19 09:03:34.393 [855955.202708] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:03:34.393 [855955.202320] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:03:34.393 [855955.201901] __writeback_single_inode+0x41/0x270 2024-10-19 09:03:34.393 [855955.201525] do_writepages+0xc9/0x230 2024-10-19 09:03:34.393 [855955.201111] ? __update_idle_core+0x23/0xb0 2024-10-19 09:03:34.393 [855955.200682] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:03:34.393 [855955.200075] write_cache_pages+0x57/0xb0 2024-10-19 09:03:34.393 [855955.199620] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:03:34.393 [855955.198958] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:03:34.393 [855955.198523] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:03:34.393 [855955.198088] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:03:34.393 [855955.197664] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:03:34.393 [855955.197189] nfs_lock_and_join_requests+0x181/0x240 [nfs] 2024-10-19 09:03:34.393 [855955.196751] ? nfs_page_group_lock+0xe/0x30 [nfs] 2024-10-19 09:03:34.393 [855955.196302] ? nfs_page_group_unlock+0x20/0x40 [nfs] 2024-10-19 09:03:34.393 [855955.195848] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:03:34.393 [855955.195528] <TASK> 2024-10-19 09:03:34.393 [855955.195189] </IRQ> 2024-10-19 09:03:34.393 [855955.194728] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:03:34.393 [855955.194283] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:03:34.393 [855955.193873] ? handle_softirqs+0x16f/0x2b0 2024-10-19 09:03:34.393 [855955.193469] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:03:34.393 [855955.193043] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:03:34.393 [855955.192626] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:03:34.393 [855955.192209] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:03:34.393 [855955.191891] <IRQ> 2024-10-19 09:03:34.393 [855955.191557] Call Trace: 2024-10-19 09:03:34.392 [855955.191201] PKRU: 55555554 2024-10-19 09:03:34.392 [855955.190685] CR2: 00007fa9b8c446c8 CR3: 000000028e822004 CR4: 0000000000770ef0 2024-10-19 09:03:34.392 [855955.190221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:03:34.392 [855955.189637] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:03:34.392 [855955.188940] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:03:34.392 [855955.188430] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 09:03:34.392 [855955.187905] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:03:34.392 [855955.187398] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636138 2024-10-19 09:03:34.392 [855955.186878] RAX: 0000000000000001 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 09:03:34.392 [855955.186445] RSP: 0018:ffffab3b4ff6b848 EFLAGS: 00000206 2024-10-19 09:03:34.392 [855955.185271] Code: 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 fb 48 8b 7f 50 48 39 df 74 05 e8 69 ff ff ff 48 8d 7b 38 f0 80 63 38 bf <48> 8b 43 38 f6 c4 10 75 06 5b c3 cc cc cc cc be 06 00 00 00 5b e9 2024-10-19 09:03:34.392 [855955.184775] RIP: 0010:nfs_page_group_unlock+0x20/0x40 [nfs] 2024-10-19 09:03:34.392 [855955.184353] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:03:34.392 [855955.183811] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:03:34.392 [855955.183394] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:03:34.392 [855955.182775] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:03:34.392 [855955.180772] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:03:34.392 [855955.176154] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:03:34.391 [855955.176119] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:03:34.391 [855955.175728] #5: 100% system, 1% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:34.391 [855955.175335] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:34.391 [855955.174928] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:34.391 [855955.174526] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:34.391 [855955.174107] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:34.391 [855955.173739] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:03:34.391 [855955.173120] watchdog: BUG: soft lockup - CPU#1 stuck for 2112s! [kworker/u24:1:21559] 2024-10-19 09:03:14.847 [ OK ] Stopped Vendor-agnostic OpenTeleme…bution with programmable pipelines. 2024-10-19 09:03:06.328 [855927.206399] </TASK> 2024-10-19 09:03:06.328 [855927.206079] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:03:06.328 [855927.205766] ? __pfx_kthread+0x10/0x10 2024-10-19 09:03:06.328 [855927.205293] ret_from_fork+0x2d/0x50 2024-10-19 09:03:06.328 [855927.204975] ? __pfx_kthread+0x10/0x10 2024-10-19 09:03:06.328 [855927.204674] kthread+0xcc/0x100 2024-10-19 09:03:06.328 [855927.204297] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:03:06.328 [855927.203967] worker_thread+0x239/0x340 2024-10-19 09:03:06.328 [855927.203618] process_one_work+0x179/0x390 2024-10-19 09:03:06.328 [855927.203282] wb_workfn+0x45/0x150 2024-10-19 09:03:06.328 [855927.202923] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:03:06.328 [855927.202567] wb_writeback+0x180/0x2f0 2024-10-19 09:03:06.328 [855927.202172] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:03:06.328 [855927.201778] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:03:06.328 [855927.201226] __writeback_single_inode+0x41/0x270 2024-10-19 09:03:06.328 [855927.200845] do_writepages+0xc9/0x230 2024-10-19 09:03:06.328 [855927.200426] ? __update_idle_core+0x23/0xb0 2024-10-19 09:03:06.328 [855927.200004] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:03:06.328 [855927.199608] write_cache_pages+0x57/0xb0 2024-10-19 09:03:06.328 [855927.199152] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:03:06.328 [855927.198683] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:03:06.328 [855927.198237] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:03:06.328 [855927.197806] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:03:06.328 [855927.197369] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:03:06.327 [855927.196906] nfs_lock_and_join_requests+0x37/0x240 [nfs] 2024-10-19 09:03:06.327 [855927.196225] nfs_folio_find_head_request+0x27/0x90 [nfs] 2024-10-19 09:03:06.327 [855927.195812] ? __pfx__raw_spin_lock+0x10/0x10 2024-10-19 09:03:06.327 [855927.195355] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:03:06.327 [855927.195033] <TASK> 2024-10-19 09:03:06.327 [855927.194707] </IRQ> 2024-10-19 09:03:06.327 [855927.194262] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:03:06.327 [855927.193816] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:03:06.327 [855927.193399] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:03:06.327 [855927.192978] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:03:06.327 [855927.192555] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:03:06.327 [855927.192133] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:03:06.327 [855927.191807] <IRQ> 2024-10-19 09:03:06.327 [855927.191457] Call Trace: 2024-10-19 09:03:06.327 [855927.191106] PKRU: 55555554 2024-10-19 09:03:06.327 [855927.190585] CR2: 00007fa9b8c446c8 CR3: 000000028e822004 CR4: 0000000000770ef0 2024-10-19 09:03:06.327 [855927.190108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:03:06.327 [855927.189554] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:03:06.327 [855927.189028] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:03:06.327 [855927.188512] R10: 000000000000003c R11: 0000000000000006 R12: 0000000000000000 2024-10-19 09:03:06.327 [855927.187990] RBP: ffff9cb2dcb84f14 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:03:06.327 [855927.187467] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9cb2dcb84f14 2024-10-19 09:03:06.327 [855927.186956] RAX: ffff9cb2dcb84e98 RBX: ffffd9d659203000 RCX: ffffffffffffffc0 2024-10-19 09:03:06.327 [855927.186521] RSP: 0018:ffffab3b4ff6b830 EFLAGS: 00000202 2024-10-19 09:03:06.327 [855927.185344] Code: b1 17 75 05 c3 cc cc cc cc e9 3c 0a 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <0f> 1f 44 00 00 65 ff 05 6c 5a 14 58 31 c0 ba 01 00 00 00 f0 0f b1 2024-10-19 09:03:06.327 [855927.184953] RIP: 0010:_raw_spin_lock+0x0/0x30 2024-10-19 09:03:06.327 [855927.184527] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:03:06.327 [855927.183985] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:03:06.327 [855927.183566] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:03:06.327 [855927.182951] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:03:06.327 [855927.180947] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:03:06.327 [855927.176366] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:03:06.327 [855927.176333] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:03:06.326 [855927.175947] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:06.326 [855927.175558] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:06.326 [855927.175165] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:06.326 [855927.174772] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:06.326 [855927.174366] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:03:06.326 [855927.174011] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:03:06.326 [855927.173416] watchdog: BUG: soft lockup - CPU#1 stuck for 2086s! [kworker/u24:1:21559] 2024-10-19 09:02:44.779 [ OK ] Unmounted /run/docker/netns/a9a06e79ab4d. 2024-10-19 09:02:44.779 [855905.711489] docker0: port 3(veth4f90e67) entered disabled state 2024-10-19 09:02:44.779 [855905.710927] veth4f90e67 (unregistering): left promiscuous mode 2024-10-19 09:02:44.779 [855905.710207] veth4f90e67 (unregistering): left allmulticast mode 2024-10-19 09:02:44.779 [855905.709126] docker0: port 3(veth4f90e67) entered disabled state 2024-10-19 09:02:44.779 [ OK ] Unmounted /run/docker/netns/bd723a8a18a5. 2024-10-19 09:02:44.779 [855905.690011] vethe214e15: renamed from eth0 2024-10-19 09:02:44.778 [855905.688657] docker0: port 3(veth4f90e67) entered disabled state 2024-10-19 09:02:44.778 [855905.678023] docker0: port 6(vethd76ed2c) entered disabled state 2024-10-19 09:02:44.778 [855905.677471] vethd76ed2c (unregistering): left promiscuous mode 2024-10-19 09:02:44.778 [855905.676897] vethd76ed2c (unregistering): left allmulticast mode 2024-10-19 09:02:44.778 [855905.675601] docker0: port 6(vethd76ed2c) entered disabled state 2024-10-19 09:02:44.778 [855905.661819] veth1ea2fc4: renamed from eth0 2024-10-19 09:02:44.778 [855905.661064] docker0: port 6(vethd76ed2c) entered disabled state 2024-10-19 09:02:38.266 [855899.207408] </TASK> 2024-10-19 09:02:38.266 [855899.207097] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:02:38.266 [855899.206735] ? __pfx_kthread+0x10/0x10 2024-10-19 09:02:38.266 [855899.206428] ret_from_fork+0x2d/0x50 2024-10-19 09:02:38.266 [855899.206113] ? __pfx_kthread+0x10/0x10 2024-10-19 09:02:38.266 [855899.205750] kthread+0xcc/0x100 2024-10-19 09:02:38.266 [855899.205407] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:02:38.266 [855899.205033] worker_thread+0x239/0x340 2024-10-19 09:02:38.266 [855899.204604] process_one_work+0x179/0x390 2024-10-19 09:02:38.266 [855899.204285] wb_workfn+0x45/0x150 2024-10-19 09:02:38.266 [855899.203936] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:02:38.266 [855899.203566] wb_writeback+0x180/0x2f0 2024-10-19 09:02:38.266 [855899.203191] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:02:38.266 [855899.202802] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:02:38.265 [855899.202371] __writeback_single_inode+0x41/0x270 2024-10-19 09:02:38.265 [855899.201989] do_writepages+0xc9/0x230 2024-10-19 09:02:38.265 [855899.201552] ? __update_idle_core+0x23/0xb0 2024-10-19 09:02:38.265 [855899.201127] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:02:38.265 [855899.200723] write_cache_pages+0x57/0xb0 2024-10-19 09:02:38.265 [855899.200276] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:02:38.265 [855899.199802] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:02:38.265 [855899.199314] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:02:38.265 [855899.198891] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:02:38.265 [855899.198458] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:02:38.265 [855899.198003] nfs_lock_and_join_requests+0x37/0x240 [nfs] 2024-10-19 09:02:38.265 [855899.197524] ? nfs_folio_find_head_request+0x27/0x90 [nfs] 2024-10-19 09:02:38.265 [855899.197055] ? nfs_folio_find_head_request+0x27/0x90 [nfs] 2024-10-19 09:02:38.265 [855899.196592] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:02:38.265 [855899.196274] <TASK> 2024-10-19 09:02:38.265 [855899.195952] </IRQ> 2024-10-19 09:02:38.265 [855899.195503] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:02:38.265 [855899.195061] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:02:38.265 [855899.194654] ? handle_softirqs+0x16f/0x2b0 2024-10-19 09:02:38.265 [855899.194248] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:02:38.265 [855899.193831] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:02:38.265 [855899.193403] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:02:38.265 [855899.192997] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:02:38.265 [855899.192673] <IRQ> 2024-10-19 09:02:38.265 [855899.192338] Call Trace: 2024-10-19 09:02:38.265 [855899.191997] PKRU: 55555554 2024-10-19 09:02:38.265 [855899.191471] CR2: 00007fa9b8c446c8 CR3: 000000028e822004 CR4: 0000000000770ef0 2024-10-19 09:02:38.265 [855899.191016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:02:38.265 [855899.190455] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:02:38.265 [855899.189946] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:02:38.265 [855899.189417] R10: 000000000000003c R11: 0000000000000006 R12: 0000000000000000 2024-10-19 09:02:38.265 [855899.188890] RBP: ffff9cb2dcb84f14 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:02:38.265 [855899.188347] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cb2dcb84f14 2024-10-19 09:02:38.265 [855899.187813] RAX: 0000000000000000 RBX: ffffd9d659203000 RCX: ffffffffffffffc0 2024-10-19 09:02:38.265 [855899.187368] RSP: 0018:ffffab3b4ff6b838 EFLAGS: 00000246 2024-10-19 09:02:38.265 [855899.186164] Code: 90 90 90 0f 1f 44 00 00 41 54 45 31 e4 55 53 48 8b 07 f6 c4 40 74 47 48 8b 47 18 48 89 fb 48 8d 68 7c 48 89 ef e8 49 d6 b0 e6 <4c> 8b 63 28 4d 85 e4 74 23 4d 39 64 24 50 75 48 49 8d 7c 24 34 b8 2024-10-19 09:02:38.265 [855899.185600] RIP: 0010:nfs_folio_find_head_request+0x27/0x90 [nfs] 2024-10-19 09:02:38.265 [855899.185182] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:02:38.265 [855899.184629] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:02:38.265 [855899.184218] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:02:38.265 [855899.183585] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:02:38.265 [855899.181292] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:02:38.264 [855899.176677] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:02:38.264 [855899.176642] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:02:38.264 [855899.176252] #5: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:38.264 [855899.175864] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:38.264 [855899.175452] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:38.264 [855899.175061] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:38.264 [855899.174652] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:38.264 [855899.174290] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:02:38.264 [855899.173712] watchdog: BUG: soft lockup - CPU#1 stuck for 2060s! [kworker/u24:1:21559] 2024-10-19 09:02:37.764 [ OK ] Stopped ElasticEndpoint. 2024-10-19 09:02:37.513 [ OK ] Stopped Enhanced C implementation …bon relay, aggregator and rewriter. 2024-10-19 09:02:37.263 [ OK ] Removed slice User Slice of UID 500. 2024-10-19 09:02:37.263 [ OK ] Stopped User Runtime Directory /run/user/500. 2024-10-19 09:02:37.263 [ OK ] Unmounted /run/user/500. 2024-10-19 09:02:37.262 Stopping User Runtime Directory /run/user/500... 2024-10-19 09:02:37.262 [ OK ] Stopped User Manager for UID 500. 2024-10-19 09:02:35.759 [ OK ] Stopped Postfix Mail Transport Agent. 2024-10-19 09:02:35.759 [ OK ] Stopped Qualys cloud agent daemon. 2024-10-19 09:02:35.759 [ OK ] Stopped C3Proxy HTTP Accelerator. 2024-10-19 09:02:35.258 [ OK ] Stopped Execmgr http worker service. 2024-10-19 09:02:35.258 [ OK ] Stopped GoodData JMX proxy. 2024-10-19 09:02:35.007 [ OK ] Stopped Agent manages other beats based on configuration provided.. 2024-10-19 09:02:34.759 Stopping User Manager for UID 5001... 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74688 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session 44907 of User rundeck. 2024-10-19 09:02:34.758 Stopping GoodData JMX proxy... 2024-10-19 09:02:34.758 Stopping Enhanced C implementation…n relay, aggregator and rewriter... 2024-10-19 09:02:34.758 Stopping C3Proxy HTTP Accelerator... 2024-10-19 09:02:34.758 [ OK ] Stopped target sshd-keygen.target. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74554 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74580 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74567 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74603 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74591 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74615 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74649 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74663 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74638 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74676 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Session c74626 of User root. 2024-10-19 09:02:34.758 [ OK ] Stopped Restore /run/initramfs on shutdown. 2024-10-19 09:02:34.758 [ OK ] Stopped GoodData Smoke testing daemon. 2024-10-19 09:02:34.758 [ OK ] Unmounted RPC Pipe File System. 2024-10-19 09:02:34.758 [ OK ] Stopped Collectd statistics daemon. 2024-10-19 09:02:34.758 [ OK ] Stopped Erlang Port Mapper Daemon. 2024-10-19 09:02:34.758 [ OK ] Stopped OpenSSH server daemon. 2024-10-19 09:02:34.758 [ OK ] Stopped C3Proxy HTTP accelerator logging daemon. 2024-10-19 09:02:34.758 [ OK ] Stopped OSSEC Agent. 2024-10-19 09:02:34.758 [ OK ] Stopped OSSEC Logcollector. 2024-10-19 09:02:34.758 [ OK ] Stopped OSSEC Execd. 2024-10-19 09:02:34.758 [ OK ] Stopped NTP client/server. 2024-10-19 09:02:34.758 [ OK ] Stopped irqbalance daemon. 2024-10-19 09:02:34.758 Stopping Vendor-agnostic OpenTelem…tion with programmable pipelines... 2024-10-19 09:02:34.758 Stopping ;[ OK ] Stopped Commit a transient machine-id on disk. 2024-10-19 09:02:34.758 Stopping Serial Getty on ttyS0... 2024-10-19 09:02:34.758 Stopping System Logging Service... 2024-10-19 09:02:34.758 Stopping Hardware RNG Entropy Gatherer Daemon... 2024-10-19 09:02:34.758 Stopping Qualys cloud agent daemon... 2024-10-19 09:02:34.758 [ OK ] Stopped Service to ensure puppet will run once on instance reboot. 2024-10-19 09:02:34.758 Stopping Postfix Mail Transport Agent... 2024-10-19 09:02:34.757 Stopping OSSEC Agent... 2024-10-19 09:02:34.757 Stopping irqbalance daemon... 2024-10-19 09:02:34.757 Stopping Getty on tty1... 2024-10-19 09:02:34.757 Stopping GoodData Compute Fabric (GCF) Worker... 2024-10-19 09:02:34.757 Stopping GoodData Compute Fabric (GCF) Proxy... 2024-10-19 09:02:34.757 Stopping Execmgr http worker service... 2024-10-19 09:02:34.757 Stopping Erlang Port Mapper Daemon... 2024-10-19 09:02:34.757 Stopping Agent manages other beats based on configuration provided.... 2024-10-19 09:02:34.757 Stopping Restore /run/initramfs on shutdown... 2024-10-19 09:02:34.757 Stopping Docker Application Container Engine... 2024-10-19 09:02:34.757 Stopping Command Scheduler... 2024-10-19 09:02:34.757 Stopping Collectd statistics daemon... 2024-10-19 09:02:34.757 [ OK ] Stopped Execute cloud user/final scripts. 2024-10-19 09:02:34.757 Stopping OSSEC syscheckd... 2024-10-19 09:02:34.757 Stopping OSSEC Logcollector... 2024-10-19 09:02:34.757 Stopping OSSEC Execd... 2024-10-19 09:02:34.757 Stopping NTP client/server... 2024-10-19 09:02:34.757 Stopping C3Proxy HTTP accelerator logging daemon... 2024-10-19 09:02:34.757 Stopping ElasticEndpoint... 2024-10-19 09:02:34.757 Unmounting RPC Pipe File System... 2024-10-19 09:02:34.757 [ OK ] Closed Process Core Dump Socket. 2024-10-19 09:02:34.757 [ OK ] Stopped daily update of the root trust anchor for DNSSEC. 2024-10-19 09:02:34.757 [ OK ] Stopped Daily Cleanup of Temporary Directories. 2024-10-19 09:02:34.757 [ OK ] Stopped Updates mlocate database every day. 2024-10-19 09:02:34.757 [ OK ] Stopped Daily rotation of log files. 2024-10-19 09:02:34.757 [ OK ] Stopped ecr-token.timer. 2024-10-19 09:02:34.757 [ OK ] Stopped alloy-gdc_snapshot.timer. 2024-10-19 09:02:34.757 [ OK ] Stopped alloy-gdc_rpm_package.timer. 2024-10-19 09:02:34.757 [ OK ] Stopped alloy-gdc_cuf.timer. 2024-10-19 09:02:34.757 [ OK ] Stopped target Timer Units. 2024-10-19 09:02:34.757 [ OK ] Stopped target RPC Port Mapper. 2024-10-19 09:02:34.757 [ OK ] Stopped target rpc_pipefs.target. 2024-10-19 09:02:34.757 [ OK ] Stopped target OSSEC HIDS agent. 2024-10-19 09:02:34.757 [ OK ] Stopped target Login Prompts. 2024-10-19 09:02:34.757 [ OK ] Stopped target Multi-User System. 2024-10-19 09:02:34.757 [ OK ] Stopped target Cloud-init target. 2024-10-19 09:02:34.757 [ OK ] Removed slice Slice /system/vault-x509. 2024-10-19 09:02:34.756 [ OK ] Removed slice Slice /system/sshd-keygen. 2024-10-19 09:02:34.756 [ OK ] Removed slice Slice /system/modprobe. 2024-10-19 09:02:34.756 Stopping Session c74688 of User root... 2024-10-19 09:02:34.756 Stopping Session c74680 of User root... 2024-10-19 09:02:34.756 Stopping Session c74676 of User root... 2024-10-19 09:02:34.756 Stopping Session c74663 of User root... 2024-10-19 09:02:34.756 Stopping Session c74649 of User root... 2024-10-19 09:02:34.756 Stopping Session c74638 of User root... 2024-10-19 09:02:34.756 Stopping Session c74626 of User root... 2024-10-19 09:02:34.756 Stopping Session c74615 of User root... 2024-10-19 09:02:34.756 Stopping Session c74603 of User root... 2024-10-19 09:02:34.756 Stopping Session c74591 of User root... 2024-10-19 09:02:34.756 Stopping Session c74580 of User root... 2024-10-19 09:02:34.756 Stopping Session c74567 of User root... 2024-10-19 09:02:34.756 Stopping Session c74554 of User root... 2024-10-19 09:02:34.756 Stopping Session 44907 of User rundeck... 2024-10-19 09:02:10.451 [855871.207928] </TASK> 2024-10-19 09:02:10.451 [855871.207610] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:02:10.451 [855871.207298] ? __pfx_kthread+0x10/0x10 2024-10-19 09:02:10.451 [855871.206976] ret_from_fork+0x2d/0x50 2024-10-19 09:02:10.451 [855871.206655] ? __pfx_kthread+0x10/0x10 2024-10-19 09:02:10.451 [855871.206355] kthread+0xcc/0x100 2024-10-19 09:02:10.451 [855871.205999] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:02:10.451 [855871.205664] worker_thread+0x239/0x340 2024-10-19 09:02:10.451 [855871.205315] process_one_work+0x179/0x390 2024-10-19 09:02:10.451 [855871.204935] wb_workfn+0x45/0x150 2024-10-19 09:02:10.451 [855871.204576] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:02:10.451 [855871.204215] wb_writeback+0x180/0x2f0 2024-10-19 09:02:10.451 [855871.203789] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:02:10.451 [855871.203393] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:02:10.451 [855871.202954] __writeback_single_inode+0x41/0x270 2024-10-19 09:02:10.451 [855871.202559] do_writepages+0xc9/0x230 2024-10-19 09:02:10.451 [855871.202109] ? __update_idle_core+0x23/0xb0 2024-10-19 09:02:10.451 [855871.201670] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:02:10.451 [855871.201252] write_cache_pages+0x57/0xb0 2024-10-19 09:02:10.451 [855871.200741] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:02:10.451 [855871.200269] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:02:10.451 [855871.199769] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:02:10.451 [855871.199321] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:02:10.451 [855871.198878] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:02:10.450 [855871.198410] nfs_lock_and_join_requests+0x37/0x240 [nfs] 2024-10-19 09:02:10.450 [855871.197885] ? nfs_folio_find_head_request+0x27/0x90 [nfs] 2024-10-19 09:02:10.450 [855871.197413] ? nfs_folio_find_head_request+0x48/0x90 [nfs] 2024-10-19 09:02:10.450 [855871.196886] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:02:10.450 [855871.196565] <TASK> 2024-10-19 09:02:10.450 [855871.196204] </IRQ> 2024-10-19 09:02:10.450 [855871.195680] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:02:10.450 [855871.195238] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:02:10.450 [855871.194817] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:02:10.450 [855871.194400] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:02:10.450 [855871.193955] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:02:10.450 [855871.193538] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:02:10.450 [855871.193186] <IRQ> 2024-10-19 09:02:10.450 [855871.192810] Call Trace: 2024-10-19 09:02:10.450 [855871.192465] PKRU: 55555554 2024-10-19 09:02:10.450 [855871.191868] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 09:02:10.450 [855871.191410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:02:10.450 [855871.190828] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:02:10.450 [855871.190314] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:02:10.450 00000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 09:02:10.199 [855871.189228] RBP: ffff9cb2dcb84f14 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:02:10.199 [855871.188650] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9ca647636134 2024-10-19 09:02:10.199 [855871.188142] RAX: 0000000000000001 RBX: ffffd9d659203000 RCX: ffffffffffffffc0 2024-10-19 09:02:10.199 [855871.187697] RSP: 0018:ffffab3b4ff6b838 EFLAGS: 00000202 2024-10-19 09:02:10.199 [855871.186526] Code: 7c 48 89 ef e8 49 d6 b0 e6 4c 8b 63 28 4d 85 e4 74 23 4d 39 64 24 50 75 48 49 8d 7c 24 34 b8 01 00 00 00 f0 41 0f c1 44 24 34 <85> c0 74 27 8d 50 01 09 c2 78 14 48 89 ef e8 05 d7 b0 e6 4c 89 e0 2024-10-19 09:02:10.199 [855871.186035] RIP: 0010:nfs_folio_find_head_request+0x48/0x90 [nfs] 2024-10-19 09:02:10.199 [855871.185613] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:02:10.199 [855871.185060] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:02:10.199 [855871.184486] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:02:10.199 [855871.183873] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:02:10.199 [855871.181851] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:02:10.199 [855871.177090] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:02:10.199 [855871.177045] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:02:10.199 [855871.176657] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:10.199 [855871.176266] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:10.199 [855871.175809] #3: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:10.199 [855871.175411] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:10.199 [855871.174985] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:02:10.199 [855871.174622] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:02:10.199 [855871.174008] watchdog: BUG: soft lockup - CPU#1 stuck for 2034s! [kworker/u24:1:21559] 2024-10-19 09:01:42.386 [855843.207721] </TASK> 2024-10-19 09:01:42.386 [855843.207397] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:01:42.386 [855843.207066] ? __pfx_kthread+0x10/0x10 2024-10-19 09:01:42.386 [855843.206755] ret_from_fork+0x2d/0x50 2024-10-19 09:01:42.386 [855843.206431] ? __pfx_kthread+0x10/0x10 2024-10-19 09:01:42.386 [855843.206078] kthread+0xcc/0x100 2024-10-19 09:01:42.386 [855843.205726] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:01:42.386 [855843.205354] worker_thread+0x239/0x340 2024-10-19 09:01:42.386 [855843.204986] process_one_work+0x179/0x390 2024-10-19 09:01:42.386 [855843.204656] wb_workfn+0x45/0x150 2024-10-19 09:01:42.386 [855843.204281] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:01:42.386 [855843.203920] wb_writeback+0x180/0x2f0 2024-10-19 09:01:42.386 [855843.203531] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:01:42.386 [855843.203114] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:01:42.386 [855843.202698] __writeback_single_inode+0x41/0x270 2024-10-19 09:01:42.386 [855843.202299] do_writepages+0xc9/0x230 2024-10-19 09:01:42.386 [855843.201877] ? __update_idle_core+0x23/0xb0 2024-10-19 09:01:42.386 [855843.201440] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:01:42.386 [855843.201024] write_cache_pages+0x57/0xb0 2024-10-19 09:01:42.386 [855843.200576] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:01:42.386 [855843.200084] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:01:42.386 [855843.199643] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:01:42.386 [855843.199186] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:01:42.386 [855843.198757] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:01:42.386 [855843.198271] nfs_lock_and_join_requests+0x181/0x240 [nfs] 2024-10-19 09:01:42.386 [855843.197824] ? nfs_page_group_lock+0xe/0x30 [nfs] 2024-10-19 09:01:42.386 [855843.197351] ? nfs_page_group_unlock+0x24/0x40 [nfs] 2024-10-19 09:01:42.386 [855843.196895] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:01:42.386 [855843.196569] <TASK> 2024-10-19 09:01:42.386 [855843.196224] </IRQ> 2024-10-19 09:01:42.386 [855843.195788] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:01:42.386 [855843.195325] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:01:42.386 [855843.194913] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:01:42.386 [855843.194490] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:01:42.386 [855843.194044] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:01:42.386 [855843.193634] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:01:42.386 [855843.193299] <IRQ> 2024-10-19 09:01:42.386 [855843.192956] Call Trace: 2024-10-19 09:01:42.385 [855843.192607] PKRU: 55555554 2024-10-19 09:01:42.385 [855843.192067] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 09:01:42.385 [855843.191602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:01:42.385 [855843.191029] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:01:42.385 [855843.190510] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:01:42.385 [855843.189974] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 09:01:42.385 [855843.189454] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:01:42.385 [855843.188921] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636138 2024-10-19 09:01:42.385 [855843.188404] RAX: 0000000000000027 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 09:01:42.385 [855843.187946] RSP: 0018:ffffab3b4ff6b848 EFLAGS: 00000206 2024-10-19 09:01:42.385 [855843.186643] Code: 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 fb 48 8b 7f 50 48 39 df 74 05 e8 69 ff ff ff 48 8d 7b 38 f0 80 63 38 bf 48 8b 43 38 <f6> c4 10 75 06 5b c3 cc cc cc cc be 06 00 00 00 5b e9 f6 df f6 e5 2024-10-19 09:01:42.385 [855843.186147] RIP: 0010:nfs_page_group_unlock+0x24/0x40 [nfs] 2024-10-19 09:01:42.385 [855843.185719] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:01:42.385 [855843.185163] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:01:42.385 [855843.184743] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:01:42.385 [855843.184118] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:01:42.385 [855843.182089] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:01:42.385 [855843.177423] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:01:42.385 [855843.177387] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:01:42.385 [855843.176976] #5: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:42.385 [855843.176580] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:42.385 [855843.176153] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:42.385 [855843.175751] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:42.385 [855843.175325] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:42.385 [855843.174950] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:01:42.385 [855843.174304] watchdog: BUG: soft lockup - CPU#1 stuck for 2008s! [kworker/u24:1:21559] 2024-10-19 09:01:14.326 [855815.224222] </TASK> 2024-10-19 09:01:14.326 [855815.223902] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:01:14.326 [855815.223570] ? __pfx_kthread+0x10/0x10 2024-10-19 09:01:14.326 [855815.223254] ret_from_fork+0x2d/0x50 2024-10-19 09:01:14.326 [855815.222932] ? __pfx_kthread+0x10/0x10 2024-10-19 09:01:14.326 [855815.222616] kthread+0xcc/0x100 2024-10-19 09:01:14.326 [855815.222247] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:01:14.326 [855815.221914] worker_thread+0x239/0x340 2024-10-19 09:01:14.326 [855815.221402] process_one_work+0x179/0x390 2024-10-19 09:01:14.326 [855815.221069] wb_workfn+0x45/0x150 2024-10-19 09:01:14.326 [855815.220696] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:01:14.326 [855815.220323] wb_writeback+0x180/0x2f0 2024-10-19 09:01:14.326 [855815.219936] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:01:14.326 [855815.219525] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:01:14.326 [855815.219109] __writeback_single_inode+0x41/0x270 2024-10-19 09:01:14.325 [855815.218722] do_writepages+0xc9/0x230 2024-10-19 09:01:14.325 [855815.218285] ? __update_idle_core+0x23/0xb0 2024-10-19 09:01:14.325 [855815.217846] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:01:14.325 [855815.217424] write_cache_pages+0x57/0xb0 2024-10-19 09:01:14.325 [855815.216969] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:01:14.325 [855815.216452] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:01:14.325 [855815.216001] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:01:14.325 [855815.215512] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:01:14.325 [855815.215033] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:01:14.325 [855815.214347] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 09:01:14.325 [855815.213536] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 09:01:14.325 [855815.212817] ? nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 09:01:14.325 [855815.212085] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:01:14.325 [855815.211567] <TASK> 2024-10-19 09:01:14.325 [855815.211060] </IRQ> 2024-10-19 09:01:14.325 [855815.210359] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:01:14.325 [855815.209616] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:01:14.325 [855815.208980] ? handle_softirqs+0x16f/0x2b0 2024-10-19 09:01:14.325 [855815.208313] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:01:14.325 [855815.207612] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:01:14.325 [855815.206940] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:01:14.325 [855815.206255] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:01:14.325 [855815.205746] <IRQ> 2024-10-19 09:01:14.325 [855815.205199] Call Trace: 2024-10-19 09:01:14.325 [855815.204626] PKRU: 55555554 2024-10-19 09:01:14.325 [855815.203785] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 09:01:14.325 [855815.203014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:01:14.325 [855815.201943] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:01:14.325 [855815.201083] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:01:14.325 [855815.200233] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 09:01:14.325 [855815.199382] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:01:14.325 [855815.198542] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 09:01:14.325 [855815.197715] RAX: 0000000000000026 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 09:01:14.325 [855815.196991] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000202 2024-10-19 09:01:14.325 [855815.195071] Code: cc cc cc cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe 48 8b 47 38 <f6> c4 20 75 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 f6 e5 66 2e 2024-10-19 09:01:14.325 [855815.194328] RIP: 0010:nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 09:01:14.324 [855815.193610] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:01:14.324 [855815.192752] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:01:14.324 [855815.192052] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:01:14.324 [855815.191051] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:01:14.324 [855815.187503] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:01:14.324 [855815.180010] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:01:14.324 [855815.179957] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:01:14.324 [855815.179298] #5: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:14.324 [855815.178599] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:14.324 [855815.177955] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:14.324 [855815.177260] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:14.324 [855815.176208] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:01:14.324 [855815.175579] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:01:14.324 [855815.174600] watchdog: BUG: soft lockup - CPU#1 stuck for 1981s! [kworker/u24:1:21559] 2024-10-19 09:00:46.266 [855787.210456] </TASK> 2024-10-19 09:00:46.266 [855787.210135] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:00:46.266 [855787.209750] ? __pfx_kthread+0x10/0x10 2024-10-19 09:00:46.266 [855787.209427] ret_from_fork+0x2d/0x50 2024-10-19 09:00:46.266 [855787.209104] ? __pfx_kthread+0x10/0x10 2024-10-19 09:00:46.266 [855787.208789] kthread+0xcc/0x100 2024-10-19 09:00:46.266 [855787.208440] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:00:46.265 [855787.208092] worker_thread+0x239/0x340 2024-10-19 09:00:46.265 [855787.207690] process_one_work+0x179/0x390 2024-10-19 09:00:46.265 [855787.207364] wb_workfn+0x45/0x150 2024-10-19 09:00:46.265 [855787.207002] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:00:46.265 [855787.206623] wb_writeback+0x180/0x2f0 2024-10-19 09:00:46.265 [855787.206236] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:00:46.265 [855787.205765] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:00:46.265 [855787.205353] __writeback_single_inode+0x41/0x270 2024-10-19 09:00:46.265 [855787.204943] do_writepages+0xc9/0x230 2024-10-19 09:00:46.265 [855787.204533] ? __update_idle_core+0x23/0xb0 2024-10-19 09:00:46.265 [855787.204099] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:00:46.265 [855787.203681] write_cache_pages+0x57/0xb0 2024-10-19 09:00:46.265 [855787.203227] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:00:46.265 [855787.202687] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:00:46.265 [855787.202242] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:00:46.265 [855787.201763] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:00:46.265 [855787.201329] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:00:46.265 [855787.200769] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 09:00:46.265 [855787.200298] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 09:00:46.265 [855787.199824] ? nfs_unlock_request+0xe/0x30 [nfs] 2024-10-19 09:00:46.265 [855787.199367] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:00:46.265 [855787.199036] <TASK> 2024-10-19 09:00:46.265 [855787.198685] </IRQ> 2024-10-19 09:00:46.265 [855787.198204] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:00:46.265 [855787.197690] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:00:46.265 [855787.197278] ? handle_softirqs+0x16f/0x2b0 2024-10-19 09:00:46.265 [855787.196674] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:00:46.265 [855787.196111] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:00:46.265 [855787.195618] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:00:46.265 [855787.195204] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:00:46.265 [855787.194809] <IRQ> 2024-10-19 09:00:46.265 [855787.194326] Call Trace: 2024-10-19 09:00:46.265 [855787.193859] PKRU: 55555554 2024-10-19 09:00:46.265 [855787.193337] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 09:00:46.265 [855787.192831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:00:46.265 [855787.192276] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:00:46.265 [855787.191673] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:00:46.265 [855787.191156] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 09:00:46.265 [855787.190616] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:00:46.264 [855787.190100] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 09:00:46.264 [855787.189567] RAX: 0000000000000000 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 09:00:46.264 [855787.189090] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000202 2024-10-19 09:00:46.264 [855787.187824] Code: 5c 41 5d c3 cc cc cc cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe <48> 8b 47 38 f6 c4 20 75 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 2024-10-19 09:00:46.264 [855787.187373] RIP: 0010:nfs_unlock_request+0xe/0x30 [nfs] 2024-10-19 09:00:46.264 [855787.186929] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:00:46.264 [855787.186391] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:00:46.264 [855787.185939] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:00:46.264 [855787.185068] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:00:46.264 [855787.182934] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:00:46.264 [855787.178140] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:00:46.264 [855787.178105] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:00:46.264 [855787.177661] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:46.264 [855787.177264] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:46.264 [855787.176812] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:46.264 [855787.176406] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:46.264 [855787.175895] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:46.264 [855787.175524] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:00:46.264 [855787.174897] watchdog: BUG: soft lockup - CPU#1 stuck for 1955s! [kworker/u24:1:21559] 2024-10-19 09:00:18.454 [855759.209540] </TASK> 2024-10-19 09:00:18.454 [855759.209146] ret_from_fork_asm+0x1a/0x30 2024-10-19 09:00:18.454 [855759.208829] ? __pfx_kthread+0x10/0x10 2024-10-19 09:00:18.454 [855759.208515] ret_from_fork+0x2d/0x50 2024-10-19 09:00:18.454 [855759.208169] ? __pfx_kthread+0x10/0x10 2024-10-19 09:00:18.454 [855759.207866] kthread+0xcc/0x100 2024-10-19 09:00:18.454 [855759.207515] ? __pfx_worker_thread+0x10/0x10 2024-10-19 09:00:18.454 [855759.207119] worker_thread+0x239/0x340 2024-10-19 09:00:18.454 [855759.206765] process_one_work+0x179/0x390 2024-10-19 09:00:18.454 [855759.206429] wb_workfn+0x45/0x150 2024-10-19 09:00:18.454 [855759.205994] wb_do_writeback+0x1cf/0x2b0 2024-10-19 09:00:18.454 [855759.205632] wb_writeback+0x180/0x2f0 2024-10-19 09:00:18.454 [855759.205163] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 09:00:18.454 [855759.204765] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 09:00:18.453 [855759.204346] __writeback_single_inode+0x41/0x270 2024-10-19 09:00:18.453 [855759.203926] do_writepages+0xc9/0x230 2024-10-19 09:00:18.453 [855759.203506] ? __update_idle_core+0x23/0xb0 2024-10-19 09:00:18.453 [855759.203032] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 09:00:18.453 [855759.202629] write_cache_pages+0x57/0xb0 2024-10-19 09:00:18.453 [855759.202158] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 09:00:18.453 [855759.201684] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 09:00:18.453 [855759.201220] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 09:00:18.453 [855759.200773] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 09:00:18.453 [855759.200303] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 09:00:18.453 [855759.199809] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 09:00:18.453 [855759.199305] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 09:00:18.453 [855759.198830] ? nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 09:00:18.453 [855759.198361] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 09:00:18.453 [855759.197987] <TASK> 2024-10-19 09:00:18.453 [855759.197661] </IRQ> 2024-10-19 09:00:18.203 [855759.197167] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 09:00:18.203 [855759.196721] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 09:00:18.203 [855759.196263] ? hrtimer_interrupt+0xff/0x240 2024-10-19 09:00:18.203 [855759.195824] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 09:00:18.203 [855759.195398] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 09:00:18.203 [855759.194938] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 09:00:18.203 [855759.194619] <IRQ> 2024-10-19 09:00:18.203 [855759.194279] Call Trace: 2024-10-19 09:00:18.203 [855759.193919] PKRU: 55555554 2024-10-19 09:00:18.203 [855759.193391] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 09:00:18.203 [855759.192872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 09:00:18.203 [855759.192223] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 09:00:18.203 [855759.191702] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 09:00:18.203 [855759.191161] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 09:00:18.203 [855759.190633] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 09:00:18.203 [855759.190067] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 09:00:18.203 [855759.189551] RAX: 0000000000000026 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 09:00:18.203 [855759.189075] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000202 2024-10-19 09:00:18.203 [855759.187813] Code: cc cc cc cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe 48 8b 47 38 <f6> c4 20 75 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 f6 e5 66 2e 2024-10-19 09:00:18.203 [855759.187359] RIP: 0010:nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 09:00:18.203 [855759.186912] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 09:00:18.203 [855759.186348] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 09:00:18.203 [855759.185886] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 09:00:18.203 [855759.185214] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 09:00:18.203 [855759.183083] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 09:00:18.203 [855759.178329] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 09:00:18.202 [855759.178293] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 09:00:18.202 [855759.177887] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:18.202 [855759.177491] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:18.202 [855759.177013] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:18.202 [855759.176611] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:18.202 [855759.176175] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 09:00:18.202 [855759.175807] CPU#1 Utilization every 4s during lockup: 2024-10-19 09:00:18.202 [855759.175191] watchdog: BUG: soft lockup - CPU#1 stuck for 1929s! [kworker/u24:1:21559] 2024-10-19 08:59:50.394 [855731.208365] </TASK> 2024-10-19 08:59:50.394 [855731.208049] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:59:50.394 [855731.207741] ? __pfx_kthread+0x10/0x10 2024-10-19 08:59:50.394 [855731.207425] ret_from_fork+0x2d/0x50 2024-10-19 08:59:50.394 [855731.207112] ? __pfx_kthread+0x10/0x10 2024-10-19 08:59:50.394 [855731.206819] kthread+0xcc/0x100 2024-10-19 08:59:50.394 [855731.206440] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:59:50.394 [855731.206114] worker_thread+0x239/0x340 2024-10-19 08:59:50.394 [855731.205771] process_one_work+0x179/0x390 2024-10-19 08:59:50.394 [855731.205429] wb_workfn+0x45/0x150 2024-10-19 08:59:50.394 [855731.205077] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:59:50.394 [855731.204727] wb_writeback+0x180/0x2f0 2024-10-19 08:59:50.394 [855731.204338] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:59:50.394 [855731.203947] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:59:50.394 [855731.203540] __writeback_single_inode+0x41/0x270 2024-10-19 08:59:50.394 [855731.203156] do_writepages+0xc9/0x230 2024-10-19 08:59:50.394 [855731.202749] ? __update_idle_core+0x23/0xb0 2024-10-19 08:59:50.394 [855731.202322] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:59:50.394 [855731.201934] write_cache_pages+0x57/0xb0 2024-10-19 08:59:50.394 [855731.201488] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:59:50.394 [855731.201025] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:59:50.394 [855731.200594] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:59:50.394 [855731.200161] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:59:50.394 [855731.199741] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:59:50.394 [855731.199275] nfs_lock_and_join_requests+0x9e/0x240 [nfs] 2024-10-19 08:59:50.394 [855731.198848] nfs_page_group_lock+0xe/0x30 [nfs] 2024-10-19 08:59:50.394 [855731.198393] ? nfs_page_set_headlock+0xc/0x60 [nfs] 2024-10-19 08:59:50.394 [855731.197954] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:59:50.394 [855731.197627] <TASK> 2024-10-19 08:59:50.394 [855731.197197] </IRQ> 2024-10-19 08:59:50.394 [855731.196600] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:59:50.393 [855731.196128] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:59:50.393 [855731.195718] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:59:50.393 [855731.195247] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:59:50.393 [855731.194641] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:59:50.393 [855731.194196] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:59:50.393 [855731.193884] <IRQ> 2024-10-19 08:59:50.393 [855731.193552] Call Trace: 2024-10-19 08:59:50.393 [855731.193200] PKRU: 55555554 2024-10-19 08:59:50.393 [855731.192693] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:59:50.393 [855731.192229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:59:50.393 [855731.191690] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:59:50.393 [855731.191167] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:59:50.393 [855731.190662] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:59:50.393 [855731.190097] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:59:50.393 [855731.189597] RDX: 0000000000000003 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 08:59:50.393 [855731.189081] RAX: ffff9ca647636100 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:59:50.393 [855731.188654] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000246 2024-10-19 08:59:50.393 [855731.187394] Code: c4 eb 89 e8 86 48 b0 e6 66 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 f0 48 0f ba 6f 38 06 <72> 07 31 c0 c3 cc cc cc cc 55 48 8d 6f 38 53 48 89 fb f0 80 4f 39 2024-10-19 08:59:50.393 [855731.186950] RIP: 0010:nfs_page_set_headlock+0xc/0x60 [nfs] 2024-10-19 08:59:50.393 [855731.186506] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:59:50.393 [855731.185985] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:59:50.393 [855731.185576] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:59:50.393 [855731.184976] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:59:50.393 [855731.183014] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:59:50.393 [855731.178438] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:59:50.393 [855731.178405] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:59:50.393 [855731.178025] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:50.393 [855731.177641] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:50.393 [855731.177244] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:50.393 [855731.176855] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:50.393 [855731.176451] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:50.393 [855731.176091] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:59:50.393 [855731.175488] watchdog: BUG: soft lockup - CPU#1 stuck for 1903s! [kworker/u24:1:21559] 2024-10-19 08:59:22.326 [855703.208724] </TASK> 2024-10-19 08:59:22.326 [855703.208404] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:59:22.326 [855703.208076] ? __pfx_kthread+0x10/0x10 2024-10-19 08:59:22.326 [855703.207761] ret_from_fork+0x2d/0x50 2024-10-19 08:59:22.326 [855703.207440] ? __pfx_kthread+0x10/0x10 2024-10-19 08:59:22.325 [855703.207139] kthread+0xcc/0x100 2024-10-19 08:59:22.325 [855703.206756] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:59:22.325 [855703.206421] worker_thread+0x239/0x340 2024-10-19 08:59:22.325 [855703.206074] process_one_work+0x179/0x390 2024-10-19 08:59:22.325 [855703.205732] wb_workfn+0x45/0x150 2024-10-19 08:59:22.325 [855703.205371] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:59:22.325 [855703.205017] wb_writeback+0x180/0x2f0 2024-10-19 08:59:22.325 [855703.204620] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:59:22.325 [855703.204229] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:59:22.325 [855703.203807] __writeback_single_inode+0x41/0x270 2024-10-19 08:59:22.325 [855703.203396] do_writepages+0xc9/0x230 2024-10-19 08:59:22.325 [855703.202992] ? __update_idle_core+0x23/0xb0 2024-10-19 08:59:22.325 [855703.202553] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:59:22.325 [855703.202164] write_cache_pages+0x57/0xb0 2024-10-19 08:59:22.325 [855703.201686] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:59:22.325 [855703.201223] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:59:22.325 [855703.200754] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:59:22.325 [855703.200138] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:59:22.325 [855703.199685] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:59:22.325 [855703.199036] ? nfs_lock_and_join_requests+0x181/0x240 [nfs] 2024-10-19 08:59:22.325 [855703.198539] ? nfs_lock_and_join_requests+0x198/0x240 [nfs] 2024-10-19 08:59:22.325 [855703.198098] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:59:22.325 [855703.197769] <TASK> 2024-10-19 08:59:22.325 [855703.197446] </IRQ> 2024-10-19 08:59:22.325 [855703.197018] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:59:22.325 [855703.196536] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:59:22.325 [855703.196138] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:59:22.325 [855703.195657] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:59:22.325 [855703.195247] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:59:22.325 [855703.194846] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:59:22.325 [855703.194520] <IRQ> 2024-10-19 08:59:22.325 [855703.194191] Call Trace: 2024-10-19 08:59:22.325 [855703.193848] PKRU: 55555554 2024-10-19 08:59:22.325 [855703.193319] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:59:22.325 [855703.192867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:59:22.325 [855703.192314] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:59:22.325 [855703.191797] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:59:22.325 [855703.191234] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:59:22.325 [855703.190394] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:59:22.325 [855703.189766] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636138 2024-10-19 08:59:22.324 [855703.189237] RAX: 0000000000000000 RBX: ffff9ca70b54d380 RCX: ffffffffffffffc0 2024-10-19 08:59:22.324 [855703.188801] RSP: 0018:ffffab3b4ff6b858 EFLAGS: 00000246 2024-10-19 08:59:22.324 [855703.187647] Code: 24 48 4d 39 e7 0f 85 48 ff ff ff 4c 89 ff e8 cf ac ff ff 45 84 f6 74 3d 49 8b 5f 48 49 39 df 0f 84 cc fe ff ff 8b 43 34 85 c0 <74> 08 48 89 df e8 9e b6 ff ff 48 8b 5b 48 49 39 df 75 e8 e9 af fe 2024-10-19 08:59:22.324 [855703.187178] RIP: 0010:nfs_lock_and_join_requests+0x198/0x240 [nfs] 2024-10-19 08:59:22.324 [855703.186752] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:59:22.324 [855703.186213] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:59:22.324 [855703.185798] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:59:22.324 [855703.185203] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:59:22.324 [855703.183252] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:59:22.324 [855703.178760] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:59:22.324 [855703.178727] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:59:22.324 [855703.178339] #5: 100% system, 1% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:22.324 [855703.177954] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:22.324 [855703.177548] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:22.324 [855703.177145] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:22.324 [855703.176742] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:59:22.324 [855703.176372] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:59:22.324 [855703.175784] watchdog: BUG: soft lockup - CPU#1 stuck for 1877s! [kworker/u24:1:21559] 2024-10-19 08:58:54.261 [855675.208355] </TASK> 2024-10-19 08:58:54.261 [855675.208036] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:58:54.261 [855675.207729] ? __pfx_kthread+0x10/0x10 2024-10-19 08:58:54.261 [855675.207427] ret_from_fork+0x2d/0x50 2024-10-19 08:58:54.261 [855675.207083] ? __pfx_kthread+0x10/0x10 2024-10-19 08:58:54.261 [855675.206791] kthread+0xcc/0x100 2024-10-19 08:58:54.261 [855675.206453] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:58:54.261 [855675.206128] worker_thread+0x239/0x340 2024-10-19 08:58:54.261 [855675.205780] process_one_work+0x179/0x390 2024-10-19 08:58:54.261 [855675.205461] wb_workfn+0x45/0x150 2024-10-19 08:58:54.261 [855675.205102] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:58:54.261 [855675.204755] wb_writeback+0x180/0x2f0 2024-10-19 08:58:54.261 [855675.204378] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:58:54.261 [855675.203978] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:58:54.261 [855675.203554] __writeback_single_inode+0x41/0x270 2024-10-19 08:58:54.261 [855675.203180] do_writepages+0xc9/0x230 2024-10-19 08:58:54.261 [855675.202766] ? __update_idle_core+0x23/0xb0 2024-10-19 08:58:54.261 [855675.202334] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:58:54.261 [855675.201914] write_cache_pages+0x57/0xb0 2024-10-19 08:58:54.261 [855675.201481] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:58:54.261 [855675.201018] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:58:54.261 [855675.200593] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:58:54.261 [855675.200173] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:58:54.261 [855675.199749] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:58:54.261 [855675.199294] ? nfs_lock_and_join_requests+0x9e/0x240 [nfs] 2024-10-19 08:58:54.261 [855675.198820] ? nfs_lock_and_join_requests+0xd4/0x240 [nfs] 2024-10-19 08:58:54.261 [855675.198381] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:58:54.261 [855675.198057] <TASK> 2024-10-19 08:58:54.261 [855675.197740] </IRQ> 2024-10-19 08:58:54.261 [855675.197309] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:58:54.261 [855675.196857] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:58:54.261 [855675.196460] ? handle_softirqs+0x16f/0x2b0 2024-10-19 08:58:54.261 [855675.196051] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:58:54.261 [855675.195635] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:58:54.261 [855675.195225] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:58:54.261 [855675.194818] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:58:54.260 [855675.194508] <IRQ> 2024-10-19 08:58:54.260 [855675.194179] Call Trace: 2024-10-19 08:58:54.260 [855675.193832] PKRU: 55555554 2024-10-19 08:58:54.260 [855675.193321] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:58:54.260 [855675.192826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:58:54.260 [855675.192287] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:58:54.260 [855675.191764] R13: ffffd9d659203000 R14: 0000000000000000 R15: ffff9ca647636100 2024-10-19 08:58:54.260 [855675.191258] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca70b54d380 2024-10-19 08:58:54.260 [855675.190709] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:58:54.260 [855675.190211] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca70b54d3b4 2024-10-19 08:58:54.260 [855675.189703] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffffffffffc0 2024-10-19 08:58:54.260 [855675.189279] RSP: 0018:ffffab3b4ff6b858 EFLAGS: 00000212 2024-10-19 08:58:54.260 [855675.188127] Code: 77 38 4d 8b 67 48 49 c1 ee 0b 41 83 e6 01 4d 39 e7 0f 84 b8 00 00 00 49 8b 44 24 38 41 8b 54 24 34 49 8d 7c 24 34 48 c1 e8 0b <83> e0 01 44 0f 45 f0 85 d2 0f 84 88 00 00 00 8d 4a 01 89 d0 f0 0f 2024-10-19 08:58:54.260 [855675.187648] RIP: 0010:nfs_lock_and_join_requests+0xd4/0x240 [nfs] 2024-10-19 08:58:54.260 [855675.187218] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:58:54.260 [855675.186685] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:58:54.260 [855675.186275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:58:54.260 [855675.185675] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:58:54.260 [855675.183719] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:58:54.260 [855675.179236] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:58:54.260 [855675.179203] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:58:54.260 [855675.178814] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:54.260 [855675.178428] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:54.260 [855675.178016] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:54.260 [855675.177486] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:54.260 [855675.177038] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:54.260 [855675.176678] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:58:54.260 [855675.176080] watchdog: BUG: soft lockup - CPU#1 stuck for 1851s! [kworker/u24:1:21559] 2024-10-19 08:58:26.449 [855647.213278] </TASK> 2024-10-19 08:58:26.449 [855647.212961] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:58:26.449 [855647.212648] ? __pfx_kthread+0x10/0x10 2024-10-19 08:58:26.449 [855647.212243] ret_from_fork+0x2d/0x50 2024-10-19 08:58:26.449 [855647.211836] ? __pfx_kthread+0x10/0x10 2024-10-19 08:58:26.449 [855647.211458] kthread+0xcc/0x100 2024-10-19 08:58:26.449 [855647.210994] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:58:26.449 [855647.210555] worker_thread+0x239/0x340 2024-10-19 08:58:26.449 [855647.210092] process_one_work+0x179/0x390 2024-10-19 08:58:26.449 [855647.209679] wb_workfn+0x45/0x150 2024-10-19 08:58:26.449 [855647.209216] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:58:26.449 [855647.208754] wb_writeback+0x180/0x2f0 2024-10-19 08:58:26.449 [855647.208249] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:58:26.449 [855647.207748] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:58:26.449 [855647.207191] __writeback_single_inode+0x41/0x270 2024-10-19 08:58:26.449 [855647.206714] do_writepages+0xc9/0x230 2024-10-19 08:58:26.449 [855647.206193] ? __update_idle_core+0x23/0xb0 2024-10-19 08:58:26.449 [855647.205669] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:58:26.449 [855647.205179] write_cache_pages+0x57/0xb0 2024-10-19 08:58:26.449 [855647.204623] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:58:26.449 [855647.204013] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:58:26.449 [855647.203463] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:58:26.449 [855647.202910] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:58:26.449 [855647.202364] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:58:26.449 [855647.201769] ? nfs_lock_and_join_requests+0x37/0x240 [nfs] 2024-10-19 08:58:26.449 [855647.201148] ? nfs_lock_and_join_requests+0x3a/0x240 [nfs] 2024-10-19 08:58:26.449 [855647.200691] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:58:26.449 [855647.200365] <TASK> 2024-10-19 08:58:26.448 [855647.200044] </IRQ> 2024-10-19 08:58:26.448 [855647.199487] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:58:26.448 [855647.198956] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:58:26.448 [855647.198556] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:58:26.448 [855647.198135] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:58:26.448 [855647.197658] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:58:26.448 [855647.197078] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:58:26.448 [855647.196713] <IRQ> 2024-10-19 08:58:26.448 [855647.196305] Call Trace: 2024-10-19 08:58:26.448 [855647.195890] PKRU: 55555554 2024-10-19 08:58:26.448 [855647.195165] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:58:26.448 [855647.194550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:58:26.448 [855647.193796] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:58:26.199 [855647.193098] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:58:26.199 [855647.192404] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:58:26.199 [855647.191718] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:58:26.199 [855647.191013] RDX: 0000000000000003 RSI: 0000000000000000 RDI: ffff9cb2dcb84f14 2024-10-19 08:58:26.199 [855647.190316] RAX: ffff9ca647636100 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:58:26.198 [855647.189729] RSP: 0018:ffffab3b4ff6b858 EFLAGS: 00000246 2024-10-19 08:58:26.198 [855647.188327] Code: 55 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 48 8b 47 18 48 8b 00 48 89 04 24 4c 89 ef e8 c9 f4 ff ff 49 89 c7 <48> 85 c0 74 2d 48 8d 58 38 eb 0c 4c 89 ff e8 53 e9 ff ff 85 c0 78 2024-10-19 08:58:26.198 [855647.187858] RIP: 0010:nfs_lock_and_join_requests+0x3a/0x240 [nfs] 2024-10-19 08:58:26.198 [855647.187439] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:58:26.198 [855647.186903] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:58:26.198 [855647.186492] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:58:26.198 [855647.185886] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:58:26.198 [855647.183899] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:58:26.198 [855647.179336] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:58:26.198 [855647.179303] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:58:26.198 [855647.178918] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:26.198 [855647.178531] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:26.198 [855647.178134] #3: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:26.198 [855647.177739] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:26.198 [855647.177336] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:58:26.198 [855647.176975] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:58:26.198 [855647.176376] watchdog: BUG: soft lockup - CPU#1 stuck for 1825s! [kworker/u24:1:21559] 2024-10-19 08:57:58.388 [855619.213699] </TASK> 2024-10-19 08:57:58.387 [855619.213381] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:57:58.387 [855619.213069] ? __pfx_kthread+0x10/0x10 2024-10-19 08:57:58.387 [855619.212759] ret_from_fork+0x2d/0x50 2024-10-19 08:57:58.387 [855619.212432] ? __pfx_kthread+0x10/0x10 2024-10-19 08:57:58.387 [855619.212134] kthread+0xcc/0x100 2024-10-19 08:57:58.387 [855619.211750] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:57:58.387 [855619.211173] worker_thread+0x239/0x340 2024-10-19 08:57:58.387 [855619.210722] process_one_work+0x179/0x390 2024-10-19 08:57:58.387 [855619.210298] wb_workfn+0x45/0x150 2024-10-19 08:57:58.387 [855619.209834] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:57:58.387 [855619.209376] wb_writeback+0x180/0x2f0 2024-10-19 08:57:58.387 [855619.208883] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:57:58.387 [855619.208368] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:57:58.387 [855619.207840] __writeback_single_inode+0x41/0x270 2024-10-19 08:57:58.387 [855619.207323] do_writepages+0xc9/0x230 2024-10-19 08:57:58.387 [855619.206811] ? __update_idle_core+0x23/0xb0 2024-10-19 08:57:58.387 [855619.206267] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:57:58.387 [855619.205774] write_cache_pages+0x57/0xb0 2024-10-19 08:57:58.387 [855619.205189] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:57:58.387 [855619.204573] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:57:58.387 [855619.204139] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:57:58.387 [855619.203701] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:57:58.387 [855619.203277] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:57:58.387 [855619.202811] nfs_lock_and_join_requests+0x181/0x240 [nfs] 2024-10-19 08:57:58.387 [855619.202373] ? nfs_page_group_lock+0xe/0x30 [nfs] 2024-10-19 08:57:58.387 [855619.201924] ? nfs_page_group_unlock+0x24/0x40 [nfs] 2024-10-19 08:57:58.387 [855619.201468] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:57:58.387 [855619.201151] <TASK> 2024-10-19 08:57:58.387 [855619.200831] </IRQ> 2024-10-19 08:57:58.387 [855619.200393] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:57:58.387 [855619.199955] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:57:58.387 [855619.199547] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:57:58.387 [855619.199132] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:57:58.387 [855619.198717] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:57:58.387 [855619.198300] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:57:58.387 [855619.197968] <IRQ> 2024-10-19 08:57:58.386 [855619.197542] Call Trace: 2024-10-19 08:57:58.386 [855619.197129] PKRU: 55555554 2024-10-19 08:57:58.386 [855619.196412] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:57:58.386 [855619.195816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:57:58.386 [855619.195068] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:57:58.386 [855619.194377] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:57:58.386 [855619.192921] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:57:58.386 [855619.191834] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:57:58.386 [855619.191075] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636138 2024-10-19 08:57:58.386 [855619.190339] RAX: 0000000000000027 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:57:58.386 [855619.189873] RSP: 0018:ffffab3b4ff6b848 EFLAGS: 00000206 2024-10-19 08:57:58.386 [855619.188685] Code: 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 fb 48 8b 7f 50 48 39 df 74 05 e8 69 ff ff ff 48 8d 7b 38 f0 80 63 38 bf 48 8b 43 38 <f6> c4 10 75 06 5b c3 cc cc cc cc be 06 00 00 00 5b e9 f6 df f6 e5 2024-10-19 08:57:58.386 [855619.188229] RIP: 0010:nfs_page_group_unlock+0x24/0x40 [nfs] 2024-10-19 08:57:58.386 [855619.187770] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:57:58.386 [855619.187232] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:57:58.386 [855619.186819] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:57:58.386 [855619.186209] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:57:58.386 [855619.184209] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:57:58.386 [855619.179633] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:57:58.386 [855619.179600] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:57:58.386 [855619.179216] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:58.386 [855619.178830] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:58.386 [855619.178431] #3: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:58.386 [855619.178039] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:58.386 [855619.177621] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:58.386 [855619.177261] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:57:58.386 [855619.176673] watchdog: BUG: soft lockup - CPU#1 stuck for 1799s! [kworker/u24:1:21559] 2024-10-19 08:57:30.328 [855591.209423] </TASK> 2024-10-19 08:57:30.328 [855591.209109] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:57:30.328 [855591.208793] ? __pfx_kthread+0x10/0x10 2024-10-19 08:57:30.328 [855591.208489] ret_from_fork+0x2d/0x50 2024-10-19 08:57:30.328 [855591.208176] ? __pfx_kthread+0x10/0x10 2024-10-19 08:57:30.328 [855591.207848] kthread+0xcc/0x100 2024-10-19 08:57:30.328 [855591.207507] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:57:30.328 [855591.207180] worker_thread+0x239/0x340 2024-10-19 08:57:30.328 [855591.206826] process_one_work+0x179/0x390 2024-10-19 08:57:30.328 [855591.206500] wb_workfn+0x45/0x150 2024-10-19 08:57:30.328 [855591.206141] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:57:30.328 [855591.205783] wb_writeback+0x180/0x2f0 2024-10-19 08:57:30.328 [855591.205401] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:57:30.328 [855591.205016] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:57:30.328 [855591.204600] __writeback_single_inode+0x41/0x270 2024-10-19 08:57:30.328 [855591.204222] do_writepages+0xc9/0x230 2024-10-19 08:57:30.328 [855591.203808] ? __update_idle_core+0x23/0xb0 2024-10-19 08:57:30.328 [855591.203388] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:57:30.328 [855591.202991] write_cache_pages+0x57/0xb0 2024-10-19 08:57:30.328 [855591.202551] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:57:30.328 [855591.202030] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:57:30.328 [855591.201590] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:57:30.328 [855591.201164] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:57:30.328 [855591.200725] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:57:30.328 [855591.200271] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 08:57:30.328 [855591.199794] ? nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 08:57:30.328 [855591.199317] ? nfs_unlock_and_release_request+0x1c/0x40 [nfs] 2024-10-19 08:57:30.328 [855591.198787] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:57:30.327 [855591.198473] <TASK> 2024-10-19 08:57:30.327 [855591.198157] </IRQ> 2024-10-19 08:57:30.327 [855591.197707] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:57:30.327 [855591.197266] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:57:30.327 [855591.196856] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:57:30.327 [855591.196422] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:57:30.327 [855591.196010] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:57:30.327 [855591.195583] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:57:30.327 [855591.195268] <IRQ> 2024-10-19 08:57:30.327 [855591.194930] Call Trace: 2024-10-19 08:57:30.327 [855591.194591] PKRU: 55555554 2024-10-19 08:57:30.327 [855591.194084] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:57:30.327 [855591.193606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:57:30.327 [855591.193065] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:57:30.327 [855591.192550] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:57:30.327 [855591.192049] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:57:30.327 [855591.191533] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:57:30.327 [855591.191031] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636134 2024-10-19 08:57:30.327 [855591.190519] RAX: 0000000000000002 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:57:30.327 [855591.190092] RSP: 0018:ffffab3b4ff6b848 EFLAGS: 00000213 2024-10-19 08:57:30.327 [855591.188928] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 fb e8 e2 fd ff ff 48 8d 7b 34 b8 ff ff ff ff f0 0f c1 43 34 <83> f8 01 74 0a 85 c0 7e 0c 5b c3 cc cc cc cc 5b e9 9f fe ff ff be 2024-10-19 08:57:30.327 [855591.188450] RIP: 0010:nfs_unlock_and_release_request+0x1c/0x40 [nfs] 2024-10-19 08:57:30.327 [855591.188035] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:57:30.327 [855591.187499] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:57:30.327 [855591.187089] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:57:30.327 [855591.186476] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:57:30.327 [855591.184514] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:57:30.327 [855591.180014] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:57:30.327 [855591.179972] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:57:30.327 [855591.179557] #5: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:30.327 [855591.179162] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:30.326 [855591.178759] #3: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:30.326 [855591.178352] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:30.326 [855591.177944] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:30.326 [855591.177574] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:57:30.326 [855591.176969] watchdog: BUG: soft lockup - CPU#1 stuck for 1773s! [kworker/u24:1:21559] 2024-10-19 08:57:02.270 [855563.211720] </TASK> 2024-10-19 08:57:02.270 [855563.211385] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:57:02.270 [855563.211064] ? __pfx_kthread+0x10/0x10 2024-10-19 08:57:02.270 [855563.210754] ret_from_fork+0x2d/0x50 2024-10-19 08:57:02.270 [855563.210438] ? __pfx_kthread+0x10/0x10 2024-10-19 08:57:02.270 [855563.210137] kthread+0xcc/0x100 2024-10-19 08:57:02.270 [855563.209735] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:57:02.270 [855563.209259] worker_thread+0x239/0x340 2024-10-19 08:57:02.270 [855563.208898] process_one_work+0x179/0x390 2024-10-19 08:57:02.270 [855563.208570] wb_workfn+0x45/0x150 2024-10-19 08:57:02.270 [855563.208182] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:57:02.270 [855563.207828] wb_writeback+0x180/0x2f0 2024-10-19 08:57:02.270 [855563.207446] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:57:02.270 [855563.207047] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:57:02.270 [855563.206640] __writeback_single_inode+0x41/0x270 2024-10-19 08:57:02.270 [855563.206253] do_writepages+0xc9/0x230 2024-10-19 08:57:02.270 [855563.205846] ? __update_idle_core+0x23/0xb0 2024-10-19 08:57:02.270 [855563.205421] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:57:02.270 [855563.204986] write_cache_pages+0x57/0xb0 2024-10-19 08:57:02.270 [855563.204546] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:57:02.270 [855563.204068] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:57:02.270 [855563.203626] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:57:02.270 [855563.203191] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:57:02.270 [855563.202768] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:57:02.270 [855563.202310] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 08:57:02.270 [855563.201842] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 08:57:02.270 [855563.201395] ? nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 08:57:02.270 [855563.200933] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:57:02.270 [855563.200615] <TASK> 2024-10-19 08:57:02.270 [855563.200297] </IRQ> 2024-10-19 08:57:02.270 [855563.199819] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:57:02.270 [855563.199307] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:57:02.270 [855563.198834] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:57:02.270 [855563.198249] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:57:02.270 [855563.197660] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:57:02.270 [855563.197080] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:57:02.270 [855563.196637] <IRQ> 2024-10-19 08:57:02.270 [855563.196149] Call Trace: 2024-10-19 08:57:02.270 [855563.195661] PKRU: 55555554 2024-10-19 08:57:02.269 [855563.194904] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:57:02.269 [855563.194421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:57:02.269 [855563.193870] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:57:02.269 [855563.193360] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:57:02.269 [855563.192847] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:57:02.269 [855563.192343] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:57:02.269 [855563.191834] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 08:57:02.269 [855563.191331] RAX: 0000000000000026 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:57:02.269 [855563.190893] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000202 2024-10-19 08:57:02.269 [855563.189739] Code: cc cc cc cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe 48 8b 47 38 <f6> c4 20 75 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 f6 e5 66 2e 2024-10-19 08:57:02.269 [855563.189297] RIP: 0010:nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 08:57:02.269 [855563.188869] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:57:02.269 [855563.188341] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:57:02.269 [855563.187886] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:57:02.269 [855563.187281] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:57:02.269 [855563.184760] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:57:02.269 [855563.180240] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:57:02.269 [855563.180207] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:57:02.269 [855563.179823] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:02.269 [855563.179434] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:02.269 [855563.179033] #3: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:02.269 [855563.178638] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:02.269 [855563.178227] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:57:02.269 [855563.177867] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:57:02.269 [855563.177265] watchdog: BUG: soft lockup - CPU#1 stuck for 1747s! [kworker/u24:1:21559] 2024-10-19 08:56:34.458 [855535.209692] </TASK> 2024-10-19 08:56:34.458 [855535.209365] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:56:34.458 [855535.209057] ? __pfx_kthread+0x10/0x10 2024-10-19 08:56:34.458 [855535.208750] ret_from_fork+0x2d/0x50 2024-10-19 08:56:34.458 [855535.208411] ? __pfx_kthread+0x10/0x10 2024-10-19 08:56:34.458 [855535.208116] kthread+0xcc/0x100 2024-10-19 08:56:34.458 [855535.207774] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:56:34.458 [855535.207439] worker_thread+0x239/0x340 2024-10-19 08:56:34.458 [855535.207096] process_one_work+0x179/0x390 2024-10-19 08:56:34.458 [855535.206771] wb_workfn+0x45/0x150 2024-10-19 08:56:34.458 [855535.206410] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:56:34.458 [855535.206059] wb_writeback+0x180/0x2f0 2024-10-19 08:56:34.458 [855535.205676] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:56:34.458 [855535.205276] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:56:34.458 [855535.204869] __writeback_single_inode+0x41/0x270 2024-10-19 08:56:34.208 [855535.204484] do_writepages+0xc9/0x230 2024-10-19 08:56:34.208 [855535.204077] ? __update_idle_core+0x23/0xb0 2024-10-19 08:56:34.208 [855535.203656] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:56:34.208 [855535.203258] write_cache_pages+0x57/0xb0 2024-10-19 08:56:34.208 [855535.202818] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:56:34.208 [855535.202347] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:56:34.208 [855535.201917] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:56:34.208 [855535.201481] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:56:34.208 [855535.201060] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:56:34.208 [855535.200603] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 08:56:34.208 [855535.200133] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 08:56:34.208 [855535.199698] ? nfs_unlock_request+0x15/0x30 [nfs] 2024-10-19 08:56:34.208 [855535.199245] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:56:34.208 [855535.198921] <TASK> 2024-10-19 08:56:34.208 [855535.198554] </IRQ> 2024-10-19 08:56:34.208 [855535.198127] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:56:34.208 [855535.197651] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:56:34.208 [855535.197229] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:56:34.208 [855535.196812] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:56:34.208 [855535.196354] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:56:34.208 [855535.195953] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:56:34.208 [855535.195641] <IRQ> 2024-10-19 08:56:34.208 [855535.195302] Call Trace: 2024-10-19 08:56:34.208 [855535.194962] PKRU: 55555554 2024-10-19 08:56:34.208 [855535.194447] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:56:34.208 [855535.193997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:56:34.208 [855535.193449] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:56:34.208 [855535.192948] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:56:34.208 [855535.192436] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:56:34.208 [855535.191937] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:56:34.208 [855535.191428] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 08:56:34.208 [855535.190929] RAX: 0000000000000026 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:56:34.208 [855535.190499] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000246 2024-10-19 08:56:34.208 [855535.189349] Code: cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe 48 8b 47 38 f6 c4 20 <75> 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 f6 e5 66 2e 0f 1f 84 2024-10-19 08:56:34.208 [855535.188915] RIP: 0010:nfs_unlock_request+0x15/0x30 [nfs] 2024-10-19 08:56:34.208 [855535.188492] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:56:34.208 [855535.187971] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:56:34.208 [855535.187543] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:56:34.207 [855535.186951] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:56:34.207 [855535.185001] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:56:34.207 [855535.180490] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:56:34.207 [855535.180458] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:56:34.207 [855535.180078] #5: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:34.207 [855535.179696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:34.207 [855535.179301] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:34.207 [855535.178912] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:34.207 [855535.178509] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:34.207 [855535.178154] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:56:34.207 [855535.177562] watchdog: BUG: soft lockup - CPU#1 stuck for 1721s! [kworker/u24:1:21559] 2024-10-19 08:56:06.400 [855507.214500] </TASK> 2024-10-19 08:56:06.400 [855507.214184] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:56:06.400 [855507.213858] ? __pfx_kthread+0x10/0x10 2024-10-19 08:56:06.400 [855507.213543] ret_from_fork+0x2d/0x50 2024-10-19 08:56:06.400 [855507.213227] ? __pfx_kthread+0x10/0x10 2024-10-19 08:56:06.400 [855507.212929] kthread+0xcc/0x100 2024-10-19 08:56:06.400 [855507.212578] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:56:06.400 [855507.212249] worker_thread+0x239/0x340 2024-10-19 08:56:06.400 [855507.211903] process_one_work+0x179/0x390 2024-10-19 08:56:06.400 [855507.211557] wb_workfn+0x45/0x150 2024-10-19 08:56:06.400 [855507.211197] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:56:06.400 [855507.210825] wb_writeback+0x180/0x2f0 2024-10-19 08:56:06.400 [855507.210425] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:56:06.400 [855507.210020] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:56:06.400 [855507.209570] __writeback_single_inode+0x41/0x270 2024-10-19 08:56:06.400 [855507.209189] do_writepages+0xc9/0x230 2024-10-19 08:56:06.400 [855507.208744] ? __update_idle_core+0x23/0xb0 2024-10-19 08:56:06.400 [855507.208324] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:56:06.400 [855507.207932] write_cache_pages+0x57/0xb0 2024-10-19 08:56:06.400 [855507.207485] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:56:06.400 [855507.207018] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:56:06.400 [855507.206576] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:56:06.400 [855507.206147] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:56:06.400 [855507.205718] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:56:06.400 [855507.205258] nfs_lock_and_join_requests+0x181/0x240 [nfs] 2024-10-19 08:56:06.400 [855507.204823] ? nfs_page_group_lock+0xe/0x30 [nfs] 2024-10-19 08:56:06.400 [855507.204377] ? nfs_page_group_unlock+0x20/0x40 [nfs] 2024-10-19 08:56:06.400 [855507.203931] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:56:06.400 [855507.203602] <TASK> 2024-10-19 08:56:06.400 [855507.203283] </IRQ> 2024-10-19 08:56:06.399 [855507.202843] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:56:06.399 [855507.202405] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:56:06.399 [855507.202005] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:56:06.399 [855507.201586] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:56:06.399 [855507.201171] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:56:06.399 [855507.200748] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:56:06.399 [855507.200432] <IRQ> 2024-10-19 08:56:06.399 [855507.200097] Call Trace: 2024-10-19 08:56:06.399 [855507.199746] PKRU: 55555554 2024-10-19 08:56:06.399 [855507.199158] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:56:06.399 [855507.198693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:56:06.399 [855507.198092] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:56:06.399 [855507.197544] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:56:06.399 [855507.196993] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:56:06.399 [855507.196480] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:56:06.399 [855507.195960] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636138 2024-10-19 08:56:06.399 [855507.195448] RAX: 0000000000000001 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:56:06.399 [855507.195014] RSP: 0018:ffffab3b4ff6b848 EFLAGS: 00000206 2024-10-19 08:56:06.399 [855507.193803] Code: 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 fb 48 8b 7f 50 48 39 df 74 05 e8 69 ff ff ff 48 8d 7b 38 f0 80 63 38 bf <48> 8b 43 38 f6 c4 10 75 06 5b c3 cc cc cc cc be 06 00 00 00 5b e9 2024-10-19 08:56:06.399 [855507.193334] RIP: 0010:nfs_page_group_unlock+0x20/0x40 [nfs] 2024-10-19 08:56:06.399 [855507.192912] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:56:06.399 [855507.192370] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:56:06.399 [855507.191956] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:56:06.399 [855507.191346] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:56:06.399 [855507.188582] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:56:06.399 [855507.181938] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:56:06.399 [855507.181905] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:56:06.399 [855507.181359] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:06.399 [855507.180806] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:06.399 [855507.180262] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:06.399 [855507.179707] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:06.399 [855507.179151] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:56:06.398 [855507.178648] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:56:06.398 [855507.177858] watchdog: BUG: soft lockup - CPU#1 stuck for 1695s! [kworker/u24:1:21559] 2024-10-19 08:55:38.335 [855479.221946] </TASK> 2024-10-19 08:55:38.335 [855479.221542] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:55:38.335 [855479.221124] ? __pfx_kthread+0x10/0x10 2024-10-19 08:55:38.335 [855479.220727] ret_from_fork+0x2d/0x50 2024-10-19 08:55:38.335 [855479.220316] ? __pfx_kthread+0x10/0x10 2024-10-19 08:55:38.335 [855479.219928] kthread+0xcc/0x100 2024-10-19 08:55:38.335 [855479.219483] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:55:38.335 [855479.219044] worker_thread+0x239/0x340 2024-10-19 08:55:38.335 [855479.218605] process_one_work+0x179/0x390 2024-10-19 08:55:38.335 [855479.218227] wb_workfn+0x45/0x150 2024-10-19 08:55:38.335 [855479.217865] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:55:38.335 [855479.217520] wb_writeback+0x180/0x2f0 2024-10-19 08:55:38.335 [855479.217129] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:55:38.335 [855479.216687] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:55:38.335 [855479.216141] __writeback_single_inode+0x41/0x270 2024-10-19 08:55:38.335 [855479.215674] do_writepages+0xc9/0x230 2024-10-19 08:55:38.335 [855479.215150] ? __update_idle_core+0x23/0xb0 2024-10-19 08:55:38.335 [855479.214600] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:55:38.335 [855479.214096] write_cache_pages+0x57/0xb0 2024-10-19 08:55:38.335 [855479.213524] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:55:38.335 [855479.212899] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:55:38.335 [855479.212346] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:55:38.335 [855479.211787] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:55:38.335 [855479.211247] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:55:38.335 [855479.210634] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 08:55:38.335 [855479.210004] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 08:55:38.335 [855479.209441] ? nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 08:55:38.335 [855479.208588] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:55:38.335 [855479.208186] <TASK> 2024-10-19 08:55:38.335 [855479.207715] </IRQ> 2024-10-19 08:55:38.335 [855479.207065] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:55:38.335 [855479.206379] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:55:38.334 [855479.205943] ? handle_softirqs+0x16f/0x2b0 2024-10-19 08:55:38.334 [855479.205498] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:55:38.334 [855479.205065] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:55:38.334 [855479.204563] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:55:38.334 [855479.204012] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:55:38.334 [855479.203690] <IRQ> 2024-10-19 08:55:38.334 [855479.203242] Call Trace: 2024-10-19 08:55:38.334 [855479.202674] PKRU: 55555554 2024-10-19 08:55:38.334 [855479.201803] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:55:38.334 [855479.201138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:55:38.334 [855479.200374] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:55:38.334 [855479.199645] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:55:38.334 [855479.198933] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:55:38.334 [855479.198236] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:55:38.334 [855479.197538] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 08:55:38.334 [855479.196829] RAX: 0000000000000026 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:55:38.334 [855479.196264] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000202 2024-10-19 08:55:38.334 [855479.194620] Code: cc cc cc cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe 48 8b 47 38 <f6> c4 20 75 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 f6 e5 66 2e 2024-10-19 08:55:38.334 [855479.194022] RIP: 0010:nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 08:55:38.334 [855479.193454] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:55:38.334 [855479.192698] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:55:38.334 [855479.192131] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:55:38.334 [855479.191278] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:55:38.334 [855479.188415] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:55:38.334 [855479.181571] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:55:38.334 [855479.181536] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:55:38.334 [855479.180912] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:38.334 [855479.180495] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:38.334 [855479.180058] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:38.334 [855479.179636] #2: 100% system, 1% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:38.334 [855479.179194] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:38.334 [855479.178773] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:55:38.334 [855479.178155] watchdog: BUG: soft lockup - CPU#1 stuck for 1669s! [kworker/u24:1:21559] 2024-10-19 08:55:10.273 [855451.211479] </TASK> 2024-10-19 08:55:10.273 [855451.211152] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:55:10.273 [855451.210846] ? __pfx_kthread+0x10/0x10 2024-10-19 08:55:10.273 [855451.210539] ret_from_fork+0x2d/0x50 2024-10-19 08:55:10.273 [855451.210215] ? __pfx_kthread+0x10/0x10 2024-10-19 08:55:10.273 [855451.209922] kthread+0xcc/0x100 2024-10-19 08:55:10.273 [855451.209579] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:55:10.273 [855451.209211] worker_thread+0x239/0x340 2024-10-19 08:55:10.273 [855451.208869] process_one_work+0x179/0x390 2024-10-19 08:55:10.273 [855451.208544] wb_workfn+0x45/0x150 2024-10-19 08:55:10.273 [855451.208181] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:55:10.273 [855451.207833] wb_writeback+0x180/0x2f0 2024-10-19 08:55:10.273 [855451.207446] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:55:10.273 [855451.207055] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:55:10.273 [855451.206652] __writeback_single_inode+0x41/0x270 2024-10-19 08:55:10.273 [855451.206255] do_writepages+0xc9/0x230 2024-10-19 08:55:10.273 [855451.205850] ? __update_idle_core+0x23/0xb0 2024-10-19 08:55:10.273 [855451.205338] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:55:10.273 [855451.204772] write_cache_pages+0x57/0xb0 2024-10-19 08:55:10.273 [855451.204161] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:55:10.273 [855451.203653] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:55:10.273 [855451.203218] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:55:10.273 [855451.202798] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:55:10.273 [855451.202370] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:55:10.273 [855451.201913] nfs_lock_and_join_requests+0x181/0x240 [nfs] 2024-10-19 08:55:10.273 [855451.201483] ? nfs_page_group_lock+0xe/0x30 [nfs] 2024-10-19 08:55:10.273 [855451.201033] ? nfs_page_group_unlock+0x24/0x40 [nfs] 2024-10-19 08:55:10.273 [855451.200593] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:55:10.273 [855451.200267] <TASK> 2024-10-19 08:55:10.273 [855451.199949] </IRQ> 2024-10-19 08:55:10.273 [855451.199489] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:55:10.273 [855451.199045] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:55:10.273 [855451.198648] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:55:10.273 [855451.198209] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:55:10.273 [855451.197801] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:55:10.273 [855451.197390] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:55:10.273 [855451.197080] <IRQ> 2024-10-19 08:55:10.273 [855451.196750] Call Trace: 2024-10-19 08:55:10.273 [855451.196400] PKRU: 55555554 2024-10-19 08:55:10.273 [855451.195896] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:55:10.273 [855451.195433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:55:10.273 [855451.194893] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:55:10.273 [855451.194373] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:55:10.273 [855451.193856] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:55:10.272 [855451.193337] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:55:10.272 [855451.192682] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636138 2024-10-19 08:55:10.272 [855451.192168] RAX: 0000000000000027 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:55:10.272 [855451.191567] RSP: 0018:ffffab3b4ff6b848 EFLAGS: 00000206 2024-10-19 08:55:10.272 [855451.190410] Code: 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 fb 48 8b 7f 50 48 39 df 74 05 e8 69 ff ff ff 48 8d 7b 38 f0 80 63 38 bf 48 8b 43 38 <f6> c4 10 75 06 5b c3 cc cc cc cc be 06 00 00 00 5b e9 f6 df f6 e5 2024-10-19 08:55:10.272 [855451.189965] RIP: 0010:nfs_page_group_unlock+0x24/0x40 [nfs] 2024-10-19 08:55:10.272 [855451.189548] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:55:10.272 [855451.189015] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:55:10.272 [855451.188603] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:55:10.272 [855451.187998] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:55:10.272 [855451.186031] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:55:10.272 [855451.181508] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:55:10.272 [855451.181473] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:55:10.272 [855451.181081] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:10.272 [855451.180696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:10.272 [855451.180294] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:10.272 [855451.179900] #2: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:10.272 [855451.179502] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:55:10.272 [855451.179129] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:55:10.272 [855451.178451] watchdog: BUG: soft lockup - CPU#1 stuck for 1642s! [kworker/u24:1:21559] 2024-10-19 08:54:42.460 [855423.213345] </TASK> 2024-10-19 08:54:42.460 [855423.213032] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:54:42.460 [855423.212715] ? __pfx_kthread+0x10/0x10 2024-10-19 08:54:42.460 [855423.212409] ret_from_fork+0x2d/0x50 2024-10-19 08:54:42.460 [855423.212095] ? __pfx_kthread+0x10/0x10 2024-10-19 08:54:42.460 [855423.211798] kthread+0xcc/0x100 2024-10-19 08:54:42.460 [855423.211444] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:54:42.460 [855423.211119] worker_thread+0x239/0x340 2024-10-19 08:54:42.460 [855423.210735] process_one_work+0x179/0x390 2024-10-19 08:54:42.460 [855423.210409] wb_workfn+0x45/0x150 2024-10-19 08:54:42.460 [855423.210057] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:54:42.460 [855423.209681] wb_writeback+0x180/0x2f0 2024-10-19 08:54:42.460 [855423.209298] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:54:42.460 [855423.208909] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:54:42.460 [855423.208477] __writeback_single_inode+0x41/0x270 2024-10-19 08:54:42.460 [855423.208101] do_writepages+0xc9/0x230 2024-10-19 08:54:42.210 [855423.207561] ? __update_idle_core+0x23/0xb0 2024-10-19 08:54:42.210 [855423.206946] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:54:42.210 [855423.206237] write_cache_pages+0x57/0xb0 2024-10-19 08:54:42.210 [855423.205495] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:54:42.210 [855423.204722] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:54:42.210 [855423.204208] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:54:42.210 [855423.203580] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:54:42.210 [855423.203109] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:54:42.210 [855423.202643] nfs_lock_and_join_requests+0x67/0x240 [nfs] 2024-10-19 08:54:42.210 [855423.202185] nfs_unlock_and_release_request+0xe/0x40 [nfs] 2024-10-19 08:54:42.210 [855423.201736] ? nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 08:54:42.210 [855423.201291] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:54:42.210 [855423.200463] <TASK> 2024-10-19 08:54:42.210 [855423.200063] </IRQ> 2024-10-19 08:54:42.210 [855423.199473] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:54:42.210 [855423.199003] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:54:42.210 [855423.198473] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:54:42.210 [855423.198052] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:54:42.210 [855423.197638] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:54:42.210 [855423.197239] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:54:42.210 [855423.196928] <IRQ> 2024-10-19 08:54:42.210 [855423.196591] Call Trace: 2024-10-19 08:54:42.210 [855423.196254] PKRU: 55555554 2024-10-19 08:54:42.210 [855423.195739] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:54:42.210 [855423.195288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:54:42.210 [855423.194741] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:54:42.210 [855423.194238] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:54:42.210 [855423.193730] R10: 000000000000003c R11: 0000000000000006 R12: ffff9ca647636100 2024-10-19 08:54:42.210 [855423.193232] RBP: ffffd9d659203000 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:54:42.210 [855423.192725] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ca647636100 2024-10-19 08:54:42.210 [855423.192227] RAX: 0000000000000026 RBX: ffff9ca647636100 RCX: ffffffffffffffc0 2024-10-19 08:54:42.210 [855423.191801] RSP: 0018:ffffab3b4ff6b840 EFLAGS: 00000202 2024-10-19 08:54:42.210 [855423.190636] Code: cc cc cc cc 0f 0b eb cb 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 4c 8d 47 38 f0 80 67 38 fe 48 8b 47 38 <f6> c4 20 75 05 c3 cc cc cc cc 31 f6 4c 89 c7 e9 2a d8 f6 e5 66 2e 2024-10-19 08:54:42.210 [855423.190201] RIP: 0010:nfs_unlock_request+0x12/0x30 [nfs] 2024-10-19 08:54:42.210 [855423.189785] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:54:42.210 [855423.189256] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:54:42.210 [855423.188847] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:54:42.210 [855423.188245] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:54:42.209 [855423.186281] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:54:42.209 [855423.181697] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:54:42.209 [855423.181663] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:54:42.209 [855423.181282] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:42.209 [855423.180898] #4: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:42.209 [855423.180500] #3: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:42.209 [855423.180108] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:42.209 [855423.179704] #1: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:42.209 [855423.179349] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:54:42.209 [855423.178748] watchdog: BUG: soft lockup - CPU#1 stuck for 1616s! [kworker/u24:1:21559] 2024-10-19 08:54:14.405 [855395.211168] </TASK> 2024-10-19 08:54:14.405 [855395.210842] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:54:14.405 [855395.210533] ? __pfx_kthread+0x10/0x10 2024-10-19 08:54:14.405 [855395.210226] ret_from_fork+0x2d/0x50 2024-10-19 08:54:14.405 [855395.209886] ? __pfx_kthread+0x10/0x10 2024-10-19 08:54:14.405 [855395.209590] kthread+0xcc/0x100 2024-10-19 08:54:14.405 [855395.209230] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:54:14.404 [855395.208897] worker_thread+0x239/0x340 2024-10-19 08:54:14.404 [855395.208554] process_one_work+0x179/0x390 2024-10-19 08:54:14.404 [855395.208228] wb_workfn+0x45/0x150 2024-10-19 08:54:14.404 [855395.207861] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:54:14.404 [855395.207510] wb_writeback+0x180/0x2f0 2024-10-19 08:54:14.404 [855395.207128] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:54:14.404 [855395.206733] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:54:14.404 [855395.206326] __writeback_single_inode+0x41/0x270 2024-10-19 08:54:14.404 [855395.205939] do_writepages+0xc9/0x230 2024-10-19 08:54:14.404 [855395.205534] ? __update_idle_core+0x23/0xb0 2024-10-19 08:54:14.404 [855395.205113] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:54:14.404 [855395.204715] write_cache_pages+0x57/0xb0 2024-10-19 08:54:14.404 [855395.204277] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:54:14.404 [855395.203804] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:54:14.404 [855395.203373] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:54:14.404 [855395.202942] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:54:14.404 [855395.202521] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:54:14.404 [855395.202066] nfs_lock_and_join_requests+0x37/0x240 [nfs] 2024-10-19 08:54:14.404 [855395.201594] nfs_folio_find_head_request+0x27/0x90 [nfs] 2024-10-19 08:54:14.404 [855395.201204] ? _raw_spin_lock+0x17/0x30 2024-10-19 08:54:14.404 [855395.200740] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:54:14.404 [855395.200425] <TASK> 2024-10-19 08:54:14.404 [855395.200106] </IRQ> 2024-10-19 08:54:14.404 [855395.199667] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:54:14.404 [855395.199226] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:54:14.404 [855395.198778] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:54:14.404 [855395.198366] ? __hrtimer_run_queues+0x10f/0x2a0 2024-10-19 08:54:14.404 [855395.197944] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-10-19 08:54:14.404 [855395.197543] ? watchdog_timer_fn+0x3ce/0x530 2024-10-19 08:54:14.404 [855395.197231] <IRQ> 2024-10-19 08:54:14.404 [855395.196892] Call Trace: 2024-10-19 08:54:14.404 [855395.196554] PKRU: 55555554 2024-10-19 08:54:14.404 [855395.196033] CR2: 00007fa9b8c446c8 CR3: 000000025fd04003 CR4: 0000000000770ef0 2024-10-19 08:54:14.404 [855395.195579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-10-19 08:54:14.404 [855395.195029] FS: 0000000000000000(0000) GS:ffff9cb3c3a80000(0000) knlGS:0000000000000000 2024-10-19 08:54:14.404 [855395.194525] R13: ffffd9d659203000 R14: 0000000000000001 R15: ffff9ca647636100 2024-10-19 08:54:14.404 [855395.194012] R10: 000000000000003c R11: 0000000000000006 R12: 0000000000000000 2024-10-19 08:54:14.404 [855395.193510] RBP: ffff9cb2dcb84f14 R08: ffff9ca647636138 R09: 0000000000000044 2024-10-19 08:54:14.404 [855395.192996] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cb2dcb84f14 2024-10-19 08:54:14.404 [855395.192497] RAX: 0000000000000000 RBX: ffffd9d659203000 RCX: ffffffffffffffc0 2024-10-19 08:54:14.404 [855395.192069] RSP: 0018:ffffab3b4ff6b830 EFLAGS: 00000246 2024-10-19 08:54:14.403 [855395.190898] Code: 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 65 ff 05 6c 5a 14 58 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e8 cb 01 00 00 90 c3 cc cc cc cc 0f 1f 2024-10-19 08:54:14.403 [855395.190510] RIP: 0010:_raw_spin_lock+0x17/0x30 2024-10-19 08:54:14.403 [855395.190090] Workqueue: writeback wb_workfn (flush-0:52) 2024-10-19 08:54:14.403 [855395.189557] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240524-3.el9 05/24/2024 2024-10-19 08:54:14.403 [855395.189144] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP 2024-10-19 08:54:14.403 [855395.188537] CPU: 1 UID: 0 PID: 21559 Comm: kworker/u24:1 Tainted: G EL 6.11.2-1.gdc.el9.x86_64 #1 2024-10-19 08:54:14.403 [855395.186566] Unloaded tainted modules: hv_vmbus(E):1 amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3 2024-10-19 08:54:14.403 [855395.182018] crct10dif_pclmul(E) ahci(E) crc32_pclmul(E) polyval_clmulni(E) libahci(E) polyval_generic(E) libata(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) 2024-10-19 08:54:14.403 [855395.181984] Modules linked in: mptcp_diag(E) xsk_diag(E) vsock_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) veth(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) vsock_loopback(E) vmw_vsock_virtio_transport_common(E) vmw_vsock_vmci_transport(E) vsock(E) vmw_vmci(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_conntrack_netlink(E) xfs(E) dm_thin_pool(E) dm_persistent_data(E) dm_bio_prison(E) dm_bufio(E) loop(E) overlay(E) dummy(E) sunrpc(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) nf_tables(E) nfnetlink(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) i2c_smbus(E) net_failover(E) vfat(E) failover(E) virtio_dma_buf(E) fat(E) virtio_balloon(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) 2024-10-19 08:54:14.403 [855395.181599] #5: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:14.403 [855395.181214] #4: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:14.403 [855395.180809] #3: 101% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:14.403 [855395.180416] #2: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:14.403 [855395.180006] #1: 100% system, 0% softirq, 0% hardirq, 0% idle 2024-10-19 08:54:14.403 [855395.179648] CPU#1 Utilization every 4s during lockup: 2024-10-19 08:54:14.403 [855395.179044] watchdog: BUG: soft lockup - CPU#1 stuck for 1590s! [kworker/u24:1:21559] 2024-10-19 08:53:46.342 [855367.211575] </TASK> 2024-10-19 08:53:46.342 [855367.211230] ret_from_fork_asm+0x1a/0x30 2024-10-19 08:53:46.342 [855367.210920] ? __pfx_kthread+0x10/0x10 2024-10-19 08:53:46.342 [855367.210615] ret_from_fork+0x2d/0x50 2024-10-19 08:53:46.342 [855367.210274] ? __pfx_kthread+0x10/0x10 2024-10-19 08:53:46.342 [855367.209977] kthread+0xcc/0x100 2024-10-19 08:53:46.342 [855367.209635] ? __pfx_worker_thread+0x10/0x10 2024-10-19 08:53:46.342 [855367.209301] worker_thread+0x239/0x340 2024-10-19 08:53:46.342 [855367.208956] process_one_work+0x179/0x390 2024-10-19 08:53:46.342 [855367.208630] wb_workfn+0x45/0x150 2024-10-19 08:53:46.342 [855367.208266] wb_do_writeback+0x1cf/0x2b0 2024-10-19 08:53:46.342 [855367.207915] wb_writeback+0x180/0x2f0 2024-10-19 08:53:46.342 [855367.207532] __writeback_inodes_wb+0x4c/0xe0 2024-10-19 08:53:46.342 [855367.207136] writeback_sb_inodes+0x21e/0x4e0 2024-10-19 08:53:46.342 [855367.206729] __writeback_single_inode+0x41/0x270 2024-10-19 08:53:46.342 [855367.206352] do_writepages+0xc9/0x230 2024-10-19 08:53:46.342 [855367.205936] ? __update_idle_core+0x23/0xb0 2024-10-19 08:53:46.342 [855367.205515] nfs_writepages+0x117/0x2b0 [nfs] 2024-10-19 08:53:46.342 [855367.205116] write_cache_pages+0x57/0xb0 2024-10-19 08:53:46.342 [855367.204677] nfs_writepages_callback+0x2d/0x50 [nfs] 2024-10-19 08:53:46.342 [855367.204203] ? __pfx_nfs_writepages_callback+0x10/0x10 [nfs] 2024-10-19 08:53:46.342 [855367.203771] nfs_page_async_flush+0x1b/0x210 [nfs] 2024-10-19 08:53:46.342 [855367.203335] ? folio_clear_dirty_for_io+0x136/0x1a0 2024-10-19 08:53:46.342 [855367.202910] ? percpu_counter_add_batch+0x51/0x90 2024-10-19 08:53:46.342 [855367.202453] nfs_lock_and_join_requests+0x37/0x240 [nfs] 2024-10-19 08:53:46.342 [855367.201982] nfs_folio_find_head_request+0x5b/0x90 [nfs] 2024-10-19 08:53:46.342 [855367.201567] ? __pfx__raw_spin_unlock+0x10/0x10 2024-10-19 08:53:46.341 [855367.201109] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 2024-10-19 08:53:46.341 [855367.200792] <TASK> 2024-10-19 08:53:46.341 [855367.200473] </IRQ> 2024-10-19 08:53:46.341 [855367.200035] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-10-19 08:53:46.341 [855367.199597] ? __sysvec_apic_timer_interrupt+0x51/0x140 2024-10-19 08:53:46.341 [855367.199185] ? hrtimer_interrupt+0xff/0x240 2024-10-19 08:53:46.341 [855367.198768] ? __hrtimer_run_queues+0x10f/0x2a0
