On Tue, Nov 20, 2012 at 10:38:45AM -0500, Josh Boyer wrote: > On Fri, Nov 16, 2012 at 3:06 PM, Mel Gorman <mgorman@xxxxxxx> wrote: > > On Fri, Nov 16, 2012 at 02:14:47PM -0500, Josh Boyer wrote: > >> On Mon, Nov 12, 2012 at 6:37 AM, Mel Gorman <mgorman@xxxxxxx> wrote: > >> > With "mm: vmscan: scale number of pages reclaimed by reclaim/compaction > >> > based on failures" reverted, Zdenek Kabelac reported the following > >> > > >> > Hmm, so it's just took longer to hit the problem and observe > >> > kswapd0 spinning on my CPU again - it's not as endless like before - > >> > but still it easily eats minutes - it helps to turn off Firefox > >> > or TB (memory hungry apps) so kswapd0 stops soon - and restart > >> > those apps again. (And I still have like >1GB of cached memory) > >> > > >> > kswapd0 R running task 0 30 2 0x00000000 > >> > ffff8801331efae8 0000000000000082 0000000000000018 0000000000000246 > >> > ffff880135b9a340 ffff8801331effd8 ffff8801331effd8 ffff8801331effd8 > >> > ffff880055dfa340 ffff880135b9a340 00000000331efad8 ffff8801331ee000 > >> > Call Trace: > >> > [<ffffffff81555bf2>] preempt_schedule+0x42/0x60 > >> > [<ffffffff81557a95>] _raw_spin_unlock+0x55/0x60 > >> > [<ffffffff81192971>] put_super+0x31/0x40 > >> > [<ffffffff81192a42>] drop_super+0x22/0x30 > >> > [<ffffffff81193b89>] prune_super+0x149/0x1b0 > >> > [<ffffffff81141e2a>] shrink_slab+0xba/0x510 > >> > > >> > The sysrq+m indicates the system has no swap so it'll never reclaim > >> > anonymous pages as part of reclaim/compaction. That is one part of the > >> > problem but not the root cause as file-backed pages could also be reclaimed. > >> > > >> > The likely underlying problem is that kswapd is woken up or kept awake > >> > for each THP allocation request in the page allocator slow path. > >> > > >> > If compaction fails for the requesting process then compaction will be > >> > deferred for a time and direct reclaim is avoided. However, if there > >> > are a storm of THP requests that are simply rejected, it will still > >> > be the the case that kswapd is awake for a prolonged period of time > >> > as pgdat->kswapd_max_order is updated each time. This is noticed by > >> > the main kswapd() loop and it will not call kswapd_try_to_sleep(). > >> > Instead it will loopp, shrinking a small number of pages and calling > >> > shrink_slab() on each iteration. > >> > > >> > The temptation is to supply a patch that checks if kswapd was woken for > >> > THP and if so ignore pgdat->kswapd_max_order but it'll be a hack and not > >> > backed up by proper testing. As 3.7 is very close to release and this is > >> > not a bug we should release with, a safer path is to revert "mm: remove > >> > __GFP_NO_KSWAPD" for now and revisit it with the view to ironing out the > >> > balance_pgdat() logic in general. > >> > > >> > Signed-off-by: Mel Gorman <mgorman@xxxxxxx> > >> > >> Does anyone know if this is queued to go into 3.7 somewhere? I looked > >> a bit and can't find it in a tree. We have a few reports of Fedora > >> rawhide users hitting this. > >> > > > > No, because I was waiting to hear if a) it worked and preferably if the > > alternative "less safe" option worked. This close to release it might be > > better to just go with the safe option. > > We've been tracking it in https://bugzilla.redhat.com/show_bug.cgi?id=866988 > and people say this revert patch doesn't seem to make the issue go away > fully. Thorsten has created another kernel with the other patch applied > for testing. > There is also a potential accounting bug that could be affecting this. https://lkml.org/lkml/2012/11/20/613 . NR_FREE_PAGES affects watermark calculations. If it's drifts too far then processes would keep entering direct reclaim and waking kswapd even if there is no need to. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>