On Wed, 22 Jul 2015, Vlastimil Babka wrote: > > I don't think the criteria for waking up khugepaged should become any more > > complex beyond its current state, which is impacted by two different > > tunables, and whether it actually has memory to scan. During this > > additional wakeup, you'd also need to pass kcompactd's node and only do > > local khugepaged scanning since there's no guarantee khugepaged can > > allocate on all nodes when one kcompactd defragments memory. > > Keeping track of the nodes where hugepage allocations are expected to succeed > is already done in this series. "local khugepaged scanning" is unfortunately > not possible in general, since the node that will be used for a given pmd is > not known until half of pte's (or more) are scanned. > When a khugepaged allocation fails for a node, it could easily kick off background compaction on that node and revisit the range later, very similar to how we can kick off background compaction in the page allocator when async or sync_light compaction fails. The distinction I'm trying to draw is between "periodic" and "background" compaction. I think there're usecases for both and we shouldn't be limiting ourselves to one or the other. Periodic compaction would wakeup at a user-defined period and fully compact memory over all nodes, round-robin at each wakeup. This keeps fragmentation low so that (ideally) background compaction or direct compaction wouldn't be needed. Background compaction would be triggered from the page allocator when async or sync_light compaction fails, regardless of whether this was from khugepaged, page fault context, or any other high-order allocation. This is an interesting discussion because I can think of lots of ways to be smart about it, but I haven't tried to implement it yet: heuristics that do ratelimiting, preemptive compaction based on fragmentation stats, etc. My rfc implements periodic compaction in khugepaged simply because we find very large thp_fault_fallback numbers and these faults tend to come in bunches so that background compaction wouldn't really help the situation itself: it's simply not fast enough and we give up compaction at fault way too early for it have a chance of being successful. I have a hard time finding other examples of that outside thp, especially at such large orders. The number one culprit that I can think of would be slub and I haven't seen any complaints about high order_fallback stats. The additional benefit of doing the periodic compaction in khugepaged is that we can do it before scanning, where alloc_sleep_millisecs is so high that kicking off background compaction on allocation failure wouldn't help. Then, storing the nodes where khugepaged allocation has failed isn't needed: the allocation itself would trigger background compaction. > > If it's unmovable fragmentation, then any periodic synchronous memory > > compaction isn't going to help either. > > It can help if it moves away movable pages out of unmovable pageblocks, so the > following unmovable allocations can be served from those pageblocks and not > fallback to pollute another movable pageblock. Even better if this is done > (kcompactd woken up) in response to such fallback, where unmovable page falls > to a partially filled movable pageblock. Stuffing also this into khugepaged > would be really a stretch. Joonsoo proposed another daemon for that in > https://lkml.org/lkml/2015/4/27/94 but extending kcompactd would be a very > natural way for this. > Sure, this is an example of why background compaction would be helpful and triggered by the page allocator when async or migrate_sync allocation fails. > > Hmm, I don't think we have to select one to the excusion of the other. I > > don't think that because khugepaged may do periodic synchronous memory > > compaction (to eventually remove direct compaction entirely from the page > > fault path, since we have checks in the page allocator that specifically > > do that) > > That would be nice for the THP page faults, yes. Or maybe just change the > default for thp "defrag" tunable to "madvise". > Right, however I'm afraid that what we have done to compaction in the fault path for MIGRATE_ASYNC has been implicitly change that default in the code :) I have examples where async compaction in the fault path scans three pageblocks and gives up because of the abort heuristics, that's not suggesting that we'll be very successful. The hope is that we can change the default to "madvise" due to periodic and background compaction and then make the "always" case do some actual defrag :) > I think pushing compaction in a workqueue would meet a bigger resistance than > new kthreads. It could be too heavyweight for this mechanism and what if > there's suddenly lots of allocations in parallel failing and scheduling the > work items? So if we do it elsewhere, I think it's best as kcompactd kthreads > and then why would we do it also in khugepaged? > We'd need the aforementioned ratelimiting to ensure that background compaction is handled appropriately, absolutely. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>