On Mon 19-11-18 18:52:02, Baoquan He wrote: [...] There are few stacks directly in the offline path but those should be OK. The real culprit seems to be the swap in code > [ +1.734416] CPU: 255 PID: 5558 Comm: stress Tainted: G L 4.20.0-rc2+ #7 > [ +0.007927] Hardware name: 9008/IT91SMUB, BIOS BLXSV512 03/22/2018 > [ +0.006297] Call Trace: > [ +0.002537] dump_stack+0x46/0x60 > [ +0.003386] __migration_entry_wait.cold.65+0x5/0x14 > [ +0.005043] do_swap_page+0x84e/0x960 > [ +0.003727] ? arch_tlb_finish_mmu+0x29/0xc0 > [ +0.006412] __handle_mm_fault+0x933/0x1330 > [ +0.004265] handle_mm_fault+0xc4/0x250 > [ +0.003915] __do_page_fault+0x2b7/0x510 > [ +0.003990] do_page_fault+0x2c/0x110 > [ +0.003729] ? page_fault+0x8/0x30 > [ +0.003462] page_fault+0x1e/0x30 There are many traces to this path. We are /* * Once page cache replacement of page migration started, page_count * *must* be zero. And, we don't want to call wait_on_page_locked() * against a page without get_page(). * So, we use get_page_unless_zero(), here. Even failed, page fault * will occur again. */ if (!get_page_unless_zero(page)) goto out; pte_unmap_unlock(ptep, ptl); wait_on_page_locked(page); taking a reference to the page under the migration. I have to think about this much more but I suspec this is just calling for a problem. Cc migration experts. For you background information. We are seeing memory offline not being able to converge because few heavily used pages fail to migrate away - e.g. http://lkml.kernel.org/r/20181116012433.GU2653@MiWiFi-R3L-srv A debugging page to dump stack for these pages http://lkml.kernel.org/r/20181116091409.GD14706@xxxxxxxxxxxxxx shows that references are taken from the swap in code (above). How are we supposed to converge when the swapin code waits for the migration to finish with the reference count elevated? -- Michal Hocko SUSE Labs
