[PATCH] mm: mglru: Fix soft lockup attributed to scanning folios

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



After we enabled mglru on our 384C1536GB production servers, we
encountered frequent soft lockups attributed to scanning folios.

The soft lockup as follows,

[Sat Feb 24 02:29:42 2024] watchdog: BUG: soft lockup - CPU#215 stuck for 111s! [kworker/215:0:2200100]
[Sat Feb 24 02:29:42 2024] Call Trace:
[Sat Feb 24 02:29:42 2024]  <IRQ>
[Sat Feb 24 02:29:42 2024]  ? show_regs.cold+0x1a/0x1f
[Sat Feb 24 02:29:42 2024]  ? watchdog_timer_fn+0x1c4/0x220
[Sat Feb 24 02:29:42 2024]  ? softlockup_fn+0x30/0x30
[Sat Feb 24 02:29:42 2024]  ? __hrtimer_run_queues+0xa2/0x2b0
[Sat Feb 24 02:29:42 2024]  ? hrtimer_interrupt+0x109/0x220
[Sat Feb 24 02:29:42 2024]  ? __sysvec_apic_timer_interrupt+0x5e/0x110
[Sat Feb 24 02:29:42 2024]  ? sysvec_apic_timer_interrupt+0x7b/0x90
[Sat Feb 24 02:29:42 2024]  </IRQ>
[Sat Feb 24 02:29:42 2024]  <TASK>
[Sat Feb 24 02:29:42 2024]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[Sat Feb 24 02:29:42 2024]  ? folio_end_writeback+0x73/0xa0
[Sat Feb 24 02:29:42 2024]  ? folio_rotate_reclaimable+0x8c/0x90
[Sat Feb 24 02:29:42 2024]  ? folio_rotate_reclaimable+0x57/0x90
[Sat Feb 24 02:29:42 2024]  ? folio_rotate_reclaimable+0x8c/0x90
[Sat Feb 24 02:29:42 2024]  folio_end_writeback+0x73/0xa0
[Sat Feb 24 02:29:42 2024]  iomap_finish_ioend+0x1d4/0x420
[Sat Feb 24 02:29:42 2024]  iomap_finish_ioends+0x5e/0xe0
[Sat Feb 24 02:29:42 2024]  xfs_end_ioend+0x65/0x150 [xfs]
[Sat Feb 24 02:29:42 2024]  xfs_end_io+0xbc/0xf0 [xfs]
[Sat Feb 24 02:29:42 2024]  process_one_work+0x1ec/0x3c0
[Sat Feb 24 02:29:42 2024]  worker_thread+0x4d/0x390
[Sat Feb 24 02:29:42 2024]  ? process_one_work+0x3c0/0x3c0
[Sat Feb 24 02:29:42 2024]  kthread+0xee/0x120
[Sat Feb 24 02:29:42 2024]  ? kthread_complete_and_exit+0x20/0x20
[Sat Feb 24 02:29:42 2024]  ret_from_fork+0x1f/0x30
[Sat Feb 24 02:29:42 2024]  </TASK>

>From our analysis of the vmcore generated by the soft lockup, the thread
was waiting for the spinlock lruvec->lru_lock:

PID: 2200100  TASK: ffff9a221d8b4000  CPU: 215  COMMAND: "kworker/215:0"
 #0 [fffffe000319ae20] crash_nmi_callback at ffffffff8e055419
 #1 [fffffe000319ae58] nmi_handle at ffffffff8e0253c0
 #2 [fffffe000319aea0] default_do_nmi at ffffffff8eae5985
 #3 [fffffe000319aec8] exc_nmi at ffffffff8eae5b78
 #4 [fffffe000319aef0] end_repeat_nmi at ffffffff8ec015f0
    [exception RIP: queued_spin_lock_slowpath+59]
    RIP: ffffffff8eaf9b8b  RSP: ffffb58b01d4fc20  RFLAGS: 00000002
    RAX: 0000000000000001  RBX: ffffb58b01d4fc90  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000001  RDI: ffff99d2b6ff9050
    RBP: ffffb58b01d4fc40   R8: 0000000000035b21   R9: 0000000000000040
    R10: 0000000000035b00  R11: 0000000000000001  R12: ffff99d2b6ff9050
    R13: 0000000000000046  R14: ffffffff8e28bd30  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
  #5 [ffffb58b01d4fc20] queued_spin_lock_slowpath at ffffffff8eaf9b8b
  #6 [ffffb58b01d4fc48] _raw_spin_lock_irqsave at ffffffff8eaf9b11
  #7 [ffffb58b01d4fc68] folio_lruvec_lock_irqsave at ffffffff8e337a82
  #8 [ffffb58b01d4fc88] folio_batch_move_lru at ffffffff8e28dbcf
  #9 [ffffb58b01d4fcd0] folio_batch_add_and_move at ffffffff8e28dce7
 #10 [ffffb58b01d4fce0] folio_rotate_reclaimable at ffffffff8e28eee7
 #11 [ffffb58b01d4fcf8] folio_end_writeback at ffffffff8e27bfb3
 #12 [ffffb58b01d4fd10] iomap_finish_ioend at ffffffff8e3d9d04
 #13 [ffffb58b01d4fd98] iomap_finish_ioends at ffffffff8e3d9fae
 #14 [ffffb58b01d4fde0] xfs_end_ioend at ffffffffc0fae835 [xfs]
 #15 [ffffb58b01d4fe20] xfs_end_io at ffffffffc0fae9dc [xfs]
 #16 [ffffb58b01d4fe60] process_one_work at ffffffff8e0ae08c
 #17 [ffffb58b01d4feb0] worker_thread at ffffffff8e0ae2ad
 #18 [ffffb58b01d4ff10] kthread at ffffffff8e0b671e
 #19 [ffffb58b01d4ff50] ret_from_fork at ffffffff8e002dcf

While the spinlock (RDI: ffff99d2b6ff9050) was held by a task which was
scanning folios:

PID: 2400713  TASK: ffff996be1d14000  CPU: 50  COMMAND: "chitu_main"
--- <NMI exception stack> ---
  #5 [ffffb58b14ef76e8] __mod_zone_page_state at ffffffff8e2a9c36
  #6 [ffffb58b14ef76f0] folio_inc_gen at ffffffff8e2990bd
  #7 [ffffb58b14ef7740] sort_folio at ffffffff8e29afbb
  #8 [ffffb58b14ef7748] sysvec_apic_timer_interrupt at ffffffff8eae79f0
  #9 [ffffb58b14ef77b0] scan_folios at ffffffff8e29b49b
 #10 [ffffb58b14ef7878] evict_folios at ffffffff8e29bb53
 #11 [ffffb58b14ef7968] lru_gen_shrink_lruvec at ffffffff8e29cb57
 #12 [ffffb58b14ef7a28] shrink_lruvec at ffffffff8e29e135
 #13 [ffffb58b14ef7af0] shrink_node at ffffffff8e29e78c
 #14 [ffffb58b14ef7b88] do_try_to_free_pages at ffffffff8e29ec08
 #15 [ffffb58b14ef7bf8] try_to_free_mem_cgroup_pages at ffffffff8e2a17a6
 #16 [ffffb58b14ef7ca8] try_charge_memcg at ffffffff8e338879
 #17 [ffffb58b14ef7d48] charge_memcg at ffffffff8e3394f8
 #18 [ffffb58b14ef7d70] __mem_cgroup_charge at ffffffff8e33aded
 #19 [ffffb58b14ef7d98] do_anonymous_page at ffffffff8e2c6523
 #20 [ffffb58b14ef7dd8] __handle_mm_fault at ffffffff8e2cc27d
 #21 [ffffb58b14ef7e78] handle_mm_fault at ffffffff8e2cc3ba
 #22 [ffffb58b14ef7eb8] do_user_addr_fault at ffffffff8e073a99
 #23 [ffffb58b14ef7f20] exc_page_fault at ffffffff8eae82f7
 #24 [ffffb58b14ef7f50] asm_exc_page_fault at ffffffff8ec00bb7

There were a total of 22 tasks waiting for this spinlock
(RDI: ffff99d2b6ff9050):

 crash> foreach RU bt | grep -B 8  queued_spin_lock_slowpath |  grep "RDI: ffff99d2b6ff9050" | wc -l
 22

Additionally, two other threads were also engaged in scanning folios, one
with 19 waiters and the other with 15 waiters.

To address this issue under heavy reclaim conditions, we introduced a
hotfix version of the fix, incorporating cond_resched() in scan_folios().
Following the application of this hotfix to our servers, the soft lockup
issue ceased.

Signed-off-by: Yafang Shao <laoar.shao@xxxxxxxxx>
Cc: Yu Zhao <yuzhao@xxxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx # 6.1+
---
 mm/vmscan.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 4f9c854ce6cc..8f2877285b9a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4367,6 +4367,10 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc,
 
 			if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH)
 				break;
+
+			spin_unlock_irq(&lruvec->lru_lock);
+			cond_resched();
+			spin_lock_irq(&lruvec->lru_lock);
 		}
 
 		if (skipped_zone) {
-- 
2.30.1 (Apple Git-130)





[Index of Archives]     [Linux Kernel]     [Kernel Development Newbies]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite Hiking]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux