The patch titled Subject: mm: take pagevecs off reclaim stack has been added to the -mm tree. Its filename is mm-take-pagevecs-off-reclaim-stack.patch Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/SubmitChecklist when testing your code *** The -mm tree is included into linux-next and is updated there every 3-4 working days ------------------------------------------------------ From: Hugh Dickins <hughd@xxxxxxxxxx> Subject: mm: take pagevecs off reclaim stack Replace pagevecs in putback_lru_pages() and move_active_pages_to_lru() by lists of pages_to_free: then apply Konstantin Khlebnikov's free_hot_cold_page_list() to them instead of pagevec_release(). Which simplifies the flow (no need to drop and retake lock whenever pagevec fills up) and reduces stale addresses in stack backtraces (which often showed through the pagevecs); but more importantly, removes another 120 bytes from the deepest stacks in page reclaim. Although I've not recently seen an actual stack overflow here with a vanilla kernel, move_active_pages_to_lru() has often featured in deep backtraces. However, free_hot_cold_page_list() does not handle compound pages (nor need it: a Transparent HugePage would have been split by the time it reaches the call in shrink_page_list()), but it is possible for putback_lru_pages() or move_active_pages_to_lru() to be left holding the last reference on a THP, so must exclude the unlikely compound case before putting on pages_to_free. Remove pagevec_strip(), its work now done in move_active_pages_to_lru(). The pagevec in scan_mapping_unevictable_pages() remains in mm/vmscan.c, but that is never on the reclaim path, and cannot be replaced by a list. Signed-off-by: Hugh Dickins <hughd@xxxxxxxxxx> Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx> Reviewed-by: Konstantin Khlebnikov <khlebnikov@xxxxxxxxxx> Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@xxxxxxxxxxxxxx> Cc: Mel Gorman <mel@xxxxxxxxx> Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> --- include/linux/pagevec.h | 2 - mm/swap.c | 19 ------------ mm/vmscan.c | 58 ++++++++++++++++++++++++++------------ 3 files changed, 40 insertions(+), 39 deletions(-) diff -puN include/linux/pagevec.h~mm-take-pagevecs-off-reclaim-stack include/linux/pagevec.h --- a/include/linux/pagevec.h~mm-take-pagevecs-off-reclaim-stack +++ a/include/linux/pagevec.h @@ -22,7 +22,6 @@ struct pagevec { void __pagevec_release(struct pagevec *pvec); void ____pagevec_lru_add(struct pagevec *pvec, enum lru_list lru); -void pagevec_strip(struct pagevec *pvec); unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping, pgoff_t start, unsigned nr_pages); unsigned pagevec_lookup_tag(struct pagevec *pvec, @@ -59,7 +58,6 @@ static inline unsigned pagevec_add(struc return pagevec_space(pvec); } - static inline void pagevec_release(struct pagevec *pvec) { if (pagevec_count(pvec)) diff -puN mm/swap.c~mm-take-pagevecs-off-reclaim-stack mm/swap.c --- a/mm/swap.c~mm-take-pagevecs-off-reclaim-stack +++ a/mm/swap.c @@ -23,7 +23,6 @@ #include <linux/init.h> #include <linux/export.h> #include <linux/mm_inline.h> -#include <linux/buffer_head.h> /* for try_to_release_page() */ #include <linux/percpu_counter.h> #include <linux/percpu.h> #include <linux/cpu.h> @@ -797,24 +796,6 @@ void ____pagevec_lru_add(struct pagevec EXPORT_SYMBOL(____pagevec_lru_add); -/* - * Try to drop buffers from the pages in a pagevec - */ -void pagevec_strip(struct pagevec *pvec) -{ - int i; - - for (i = 0; i < pagevec_count(pvec); i++) { - struct page *page = pvec->pages[i]; - - if (page_has_private(page) && trylock_page(page)) { - if (page_has_private(page)) - try_to_release_page(page, 0); - unlock_page(page); - } - } -} - /** * pagevec_lookup - gang pagecache lookup * @pvec: Where the resulting pages are placed diff -puN mm/vmscan.c~mm-take-pagevecs-off-reclaim-stack mm/vmscan.c --- a/mm/vmscan.c~mm-take-pagevecs-off-reclaim-stack +++ a/mm/vmscan.c @@ -1398,12 +1398,10 @@ putback_lru_pages(struct mem_cgroup_zone struct list_head *page_list) { struct page *page; - struct pagevec pvec; + LIST_HEAD(pages_to_free); struct zone *zone = mz->zone; struct zone_reclaim_stat *reclaim_stat = get_reclaim_stat(mz); - pagevec_init(&pvec, 1); - /* * Put back any unfreeable pages. */ @@ -1438,17 +1436,24 @@ putback_lru_pages(struct mem_cgroup_zone int numpages = hpage_nr_pages(page); reclaim_stat->recent_rotated[file] += numpages; } - if (!pagevec_add(&pvec, page)) { - spin_unlock_irq(&zone->lru_lock); - __pagevec_release(&pvec); - spin_lock_irq(&zone->lru_lock); + if (put_page_testzero(page)) { + __ClearPageLRU(page); + __ClearPageActive(page); + del_page_from_lru_list(zone, page, lru); + + if (unlikely(PageCompound(page))) { + spin_unlock_irq(&zone->lru_lock); + (*get_compound_page_dtor(page))(page); + spin_lock_irq(&zone->lru_lock); + } else + list_add(&page->lru, &pages_to_free); } } __mod_zone_page_state(zone, NR_ISOLATED_ANON, -nr_anon); __mod_zone_page_state(zone, NR_ISOLATED_FILE, -nr_file); spin_unlock_irq(&zone->lru_lock); - pagevec_release(&pvec); + free_hot_cold_page_list(&pages_to_free, 1); } static noinline_for_stack void @@ -1658,13 +1663,23 @@ shrink_inactive_list(unsigned long nr_to static void move_active_pages_to_lru(struct zone *zone, struct list_head *list, + struct list_head *pages_to_free, enum lru_list lru) { unsigned long pgmoved = 0; - struct pagevec pvec; struct page *page; - pagevec_init(&pvec, 1); + if (buffer_heads_over_limit) { + spin_unlock_irq(&zone->lru_lock); + list_for_each_entry(page, list, lru) { + if (page_has_private(page) && trylock_page(page)) { + if (page_has_private(page)) + try_to_release_page(page, 0); + unlock_page(page); + } + } + spin_lock_irq(&zone->lru_lock); + } while (!list_empty(list)) { struct lruvec *lruvec; @@ -1678,12 +1693,17 @@ static void move_active_pages_to_lru(str list_move(&page->lru, &lruvec->lists[lru]); pgmoved += hpage_nr_pages(page); - if (!pagevec_add(&pvec, page) || list_empty(list)) { - spin_unlock_irq(&zone->lru_lock); - if (buffer_heads_over_limit) - pagevec_strip(&pvec); - __pagevec_release(&pvec); - spin_lock_irq(&zone->lru_lock); + if (put_page_testzero(page)) { + __ClearPageLRU(page); + __ClearPageActive(page); + del_page_from_lru_list(zone, page, lru); + + if (unlikely(PageCompound(page))) { + spin_unlock_irq(&zone->lru_lock); + (*get_compound_page_dtor(page))(page); + spin_lock_irq(&zone->lru_lock); + } else + list_add(&page->lru, pages_to_free); } } __mod_zone_page_state(zone, NR_LRU_BASE + lru, pgmoved); @@ -1777,12 +1797,14 @@ static void shrink_active_list(unsigned */ reclaim_stat->recent_rotated[file] += nr_rotated; - move_active_pages_to_lru(zone, &l_active, + move_active_pages_to_lru(zone, &l_active, &l_hold, LRU_ACTIVE + file * LRU_FILE); - move_active_pages_to_lru(zone, &l_inactive, + move_active_pages_to_lru(zone, &l_inactive, &l_hold, LRU_BASE + file * LRU_FILE); __mod_zone_page_state(zone, NR_ISOLATED_ANON + file, -nr_taken); spin_unlock_irq(&zone->lru_lock); + + free_hot_cold_page_list(&l_hold, 1); } #ifdef CONFIG_SWAP _ Subject: Subject: mm: take pagevecs off reclaim stack Patches currently in -mm which might be from hughd@xxxxxxxxxx are linux-next.patch mm-add-free_hot_cold_page_list-helper.patch mm-add-free_hot_cold_page_list-helper-v2.patch mm-add-free_hot_cold_page_list-helper-v3.patch mm-remove-unused-pagevec_free.patch mm-tracepoint-rename-page-free-events.patch mm-tracepoint-fixup-documentation-and-examples.patch mremap-enforce-rmap-src-dst-vma-ordering-in-case-of-vma_merge-succeeding-in-copy_vma.patch mremap-enforce-rmap-src-dst-vma-ordering-in-case-of-vma_merge-succeeding-in-copy_vma-update.patch mm-simplify-find_vma_prev.patch mm-test-pageswapbacked-in-lumpy-reclaim.patch memcg-add-mem_cgroup_replace_page_cache-to-fix-lru-issue.patch mm-memcg-consolidate-hierarchy-iteration-primitives.patch mm-vmscan-distinguish-global-reclaim-from-global-lru-scanning.patch mm-vmscan-distinguish-between-memcg-triggering-reclaim-and-memcg-being-scanned.patch mm-memcg-per-priority-per-zone-hierarchy-scan-generations.patch mm-move-memcg-hierarchy-reclaim-to-generic-reclaim-code.patch mm-memcg-remove-optimization-of-keeping-the-root_mem_cgroup-lru-lists-empty.patch mm-vmscan-convert-global-reclaim-to-per-memcg-lru-lists.patch mm-collect-lru-list-heads-into-struct-lruvec.patch mm-make-per-memcg-lru-lists-exclusive.patch mm-memcg-remove-unused-node-section-info-from-pc-flags.patch mm-memcg-remove-unused-node-section-info-from-pc-flags-fix.patch mm-oom_kill-remove-memcg-argument-from-oom_kill_task.patch mm-unify-remaining-mem_cont-mem-etc-variable-names-to-memcg.patch mm-memcg-clean-up-fault-accounting.patch mm-memcg-lookup_page_cgroup-almost-never-returns-null.patch mm-memcg-lookup_page_cgroup-almost-never-returns-null-fix.patch mm-page_cgroup-check-page_cgroup-arrays-in-lookup_page_cgroup-only-when-necessary.patch mm-memcg-remove-unneeded-checks-from-newpage_charge.patch mm-memcg-remove-unneeded-checks-from-uncharge_page.patch memcg-clean-up-soft_limit_tree-if-allocation-fails.patch memcg-simplify-page-cache-charging.patch memcg-simplify-corner-case-handling-of-lru.patch memcg-clear-pc-mem_cgorup-if-necessary.patch memcg-clear-pc-mem_cgorup-if-necessary-fix.patch memcg-clear-pc-mem_cgorup-if-necessary-fix-2.patch memcg-clear-pc-mem_cgorup-if-necessary-fix-2-fix.patch memcg-clear-pc-mem_cgorup-if-necessary-fix-3.patch memcg-clear-pc-mem_cgorup-if-necessary-fix-page-migration-to-reset_owner.patch memcg-simplify-lru-handling-by-new-rule.patch memcg-simplify-lru-handling-by-new-rule-fix.patch memcg-simplify-lru-handling-by-new-rule-memcg-return-eintr-at-bypassing-try_charge.patch memcg-simplify-lru-handling-by-new-rule-memcg-return-eintr-at-bypassing-try_charge-fix.patch memcg-simplify-lru-handling-by-new-rule-memcg-return-eintr-at-bypassing-try_charge-fix-null-mem_cgroup_try_charge.patch memcg-fix-split_huge_page_refcounts.patch memcg-fix-mem_cgroup_print_bad_page.patch mm-take-pagevecs-off-reclaim-stack.patch procfs-add-hidepid=-and-gid=-mount-options-fix.patch radix_tree-remove-radix_tree_indirect_to_ptr.patch radix_tree-take-radix_tree_path-off-stack.patch radix_tree-take-radix_tree_path-off-stack-expand-comment-on-optimization.patch prio_tree-debugging-patch.patch -- To unsubscribe from this list: send the line "unsubscribe mm-commits" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html