On 04.07.24 09:43, Zhijian Li (Fujitsu) wrote:
All,
Some progress updates
When issue occurs, calling __drain_all_pages() can make offline_pages() escape from the loop.
Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:29:26 linux kernel: page dumped because: trouble page...
With this problematic page structure contents, it seems that the
list_head = {ffffdfbd9e603788, ffffd4f0ffd97ef0} is valid.
I guess it was linking to the pcp_list, so I dumped the per_cpu_pages[cpu].count
in every in critical timings.
So, is your reproducer getting fixed when you call __drain_all_pages()
in the loop? (not that it's the right fix, but a could datapoint :) )
An example is as below,
offline_pages()
{
// per_cpu_pages[1].count = 0
zone_pcp_disable() // will call __drain_all_pages()
// per_cpu_pages[1].count = 188
do {
do {
scan_movable_pages()
ret = do_migrate_range()
} while (!ret)
ret = test_pages_isolated()
if(is the 1st iteration)
// per_cpu_pages[1].count = 182
if (issue occurs) { /* if the loop take beyond 10 seconds */
// per_cpu_pages[1].count = 61
__drain_all_pages()
// per_cpu_pages[1].count = 0
/* will escape from the outer loop in later iterations */
}
} while (ret)
}
Some interesting points:
- After the 1st __drain_all_pages(), per_cpu_pages[1].count increased to 188 from 0,
does it mean it's racing with something...?
- per_cpu_pages[1].count will decrease but not decrease to 0 during iterations
- when issue occurs, calling __drain_all_pages() will decrease per_cpu_pages[1].count to 0.
That's indeed weird. Maybe there is a race, or zone_pcp_disable() is not
fully effective for a zone?
So I wonder if it's fine to call __drain_all_pages() in the loop?
Looking forward to your insights.
So, in free_unref_page(), we make sure to never place MIGRATE_ISOLATE
onto the PCP. All pageblocks we are going to offline should be isolated
at this point, so no page that is getting freed and part of the
to-be-offlined range should end up on the PCP. So far the theory.
In offlining code we do
1) Set MIGRATE_ISOLATE
2) zone_pcp_disable() -> set high-and-batch to 0 and drain
Could there be a race in free_unref_page(), such that although
zone_pcp_disable() succeeds, we would still end up with a page in the
pcp? (especially, one that has MIGRATE_ISOLATE set for its pageblock?)
--
Cheers,
David / dhildenb
