The patch titled Subject: mm: don't use compound_head() in virt_to_head_page() has been removed from the -mm tree. Its filename was mm-dont-use-compound_head-in-virt_to_head_page.patch This patch was dropped because it was merged into mainline or a subsystem tree ------------------------------------------------------ From: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx> Subject: mm: don't use compound_head() in virt_to_head_page() compound_head() is implemented with assumption that there would be race condition when checking tail flag. This assumption is only true when we try to access arbitrary positioned struct page. The situation that virt_to_head_page() is called is different case. We call virt_to_head_page() only in the range of allocated pages, so there is no race condition on tail flag. In this case, we don't need to handle race condition and we can reduce overhead slightly. This patch implements compound_head_fast() which is similar with compound_head() except tail flag race handling. And then, virt_to_head_page() uses this optimized function to improve performance. I saw 1.8% win in a fast-path loop over kmem_cache_alloc/free, (14.063 ns -> 13.810 ns) if target object is on tail page. Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx> Acked-by: Christoph Lameter <cl@xxxxxxxxx> Cc: Pekka Enberg <penberg@xxxxxxxxxx> Cc: David Rientjes <rientjes@xxxxxxxxxx> Cc: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx> Cc: Jesper Dangaard Brouer <brouer@xxxxxxxxxx> Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> --- include/linux/mm.h | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff -puN include/linux/mm.h~mm-dont-use-compound_head-in-virt_to_head_page include/linux/mm.h --- a/include/linux/mm.h~mm-dont-use-compound_head-in-virt_to_head_page +++ a/include/linux/mm.h @@ -454,6 +454,12 @@ static inline struct page *compound_head return tail; } +/* + * Since either compound page could be dismantled asynchronously in THP + * or we access asynchronously arbitrary positioned struct page, there + * would be tail flag race. To handle this race, we should call + * smp_rmb() before checking tail flag. compound_head_by_tail() did it. + */ static inline struct page *compound_head(struct page *page) { if (unlikely(PageTail(page))) @@ -462,6 +468,18 @@ static inline struct page *compound_head } /* + * If we access compound page synchronously such as access to + * allocated page, there is no need to handle tail flag race, so we can + * check tail flag directly without any synchronization primitive. + */ +static inline struct page *compound_head_fast(struct page *page) +{ + if (unlikely(PageTail(page))) + return page->first_page; + return page; +} + +/* * The atomic page->_mapcount, starts from -1: so that transitions * both from it and to it can be tracked, using atomic_inc_and_test * and atomic_add_negative(-1). @@ -539,7 +557,14 @@ static inline void get_page(struct page static inline struct page *virt_to_head_page(const void *x) { struct page *page = virt_to_page(x); - return compound_head(page); + + /* + * We don't need to worry about synchronization of tail flag + * when we call virt_to_head_page() since it is only called for + * already allocated page and this page won't be freed until + * this virt_to_head_page() is finished. So use _fast variant. + */ + return compound_head_fast(page); } /* _ Patches currently in -mm which might be from iamjoonsoo.kim@xxxxxxx are origin.patch sparc32-fix-broken-set_pte.patch mm-set-page-pfmemalloc-in-prep_new_page.patch mm-page_alloc-reduce-number-of-alloc_pages-functions-parameters.patch mm-reduce-try_to_compact_pages-parameters.patch mm-microoptimize-zonelist-operations.patch list_lru-introduce-list_lru_shrink_countwalk.patch fs-consolidate-nrfree_cached_objects-args-in-shrink_control.patch vmscan-per-memory-cgroup-slab-shrinkers.patch memcg-rename-some-cache-id-related-variables.patch memcg-add-rwsem-to-synchronize-against-memcg_caches-arrays-relocation.patch list_lru-get-rid-of-active_nodes.patch list_lru-organize-all-list_lrus-to-list.patch list_lru-introduce-per-memcg-lists.patch fs-make-shrinker-memcg-aware.patch mm-cma-fix-totalcma_pages-to-include-dt-defined-cma-regions.patch mm-compaction-change-tracepoint-format-from-decimal-to-hexadecimal.patch mm-compaction-enhance-tracepoint-output-for-compaction-begin-end.patch mm-compaction-enhance-tracepoint-output-for-compaction-begin-end-v4.patch mm-compaction-enhance-tracepoint-output-for-compaction-begin-end-v4-fix.patch mm-compaction-print-current-range-where-compaction-work.patch mm-compaction-more-trace-to-understand-when-why-compaction-start-finish.patch mm-compaction-add-tracepoint-to-observe-behaviour-of-compaction-defer.patch mm-compaction-add-tracepoint-to-observe-behaviour-of-compaction-defer-v4.patch slab-embed-memcg_cache_params-to-kmem_cache.patch slab-link-memcg-caches-of-the-same-kind-into-a-list.patch cgroup-release-css-id-after-css_free.patch slab-use-css-id-for-naming-per-memcg-caches.patch memcg-free-memcg_caches-slot-on-css-offline.patch list_lru-add-helpers-to-isolate-items.patch memcg-reparent-list_lrus-and-free-kmemcg_id-on-css-offline.patch slub-never-fail-to-shrink-cache.patch slub-fix-kmem_cache_shrink-return-value.patch slub-make-dead-caches-discard-free-slabs-immediately.patch mm-when-stealing-freepages-also-take-pages-created-by-splitting-buddy-page.patch mm-always-steal-split-buddies-in-fallback-allocations.patch mm-more-aggressive-page-stealing-for-unmovable-allocations.patch mm-page_ext-remove-unnecessary-stack_trace-field.patch mm-page_ext-remove-unnecessary-stack_trace-field-fix.patch mm-compaction-fix-wrong-order-check-in-compact_finished.patch mm-compaction-stop-the-isolation-when-we-isolate-enough-freepage.patch mm-internalh-dont-split-printk-call-in-two.patch mm-page_allocc-pull-out-init-code-from-build_all_zonelists.patch mm-mm_initc-mark-mminit_verify_zonelist-as-__init.patch mm-mm_initc-mark-mminit_loglevel-__meminitdata.patch kernel-cpusetc-mark-cpuset_init_current_mems_allowed-as-__init.patch mm-util-add-kstrdup_const.patch kernfs-convert-node-name-allocation-to-kstrdup_const.patch clk-convert-clock-name-allocations-to-kstrdup_const.patch mm-slab-convert-cache-name-allocations-to-kstrdup_const.patch mm-slab-convert-cache-name-allocations-to-kstrdup_const-fix.patch fs-namespace-convert-devname-allocation-to-kstrdup_const.patch compiler-introduce-__aliassymbol-shortcut.patch add-kernel-address-sanitizer-infrastructure.patch kasan-disable-memory-hotplug.patch x86_64-add-kasan-support.patch mm-page_alloc-add-kasan-hooks-on-alloc-and-free-paths.patch mm-slub-introduce-virt_to_obj-function.patch mm-slub-share-object_err-function.patch mm-slub-introduce-metadata_access_enable-metadata_access_disable.patch mm-slub-add-kernel-address-sanitizer-support-for-slub-allocator.patch fs-dcache-manually-unpoison-dname-after-allocation-to-shut-up-kasans-reports.patch kmemleak-disable-kasan-instrumentation-for-kmemleak.patch lib-add-kasan-test-module.patch x86_64-kasan-add-interceptors-for-memset-memmove-memcpy-functions.patch kasan-enable-stack-instrumentation.patch mm-vmalloc-add-flag-preventing-guard-hole-allocation.patch mm-vmalloc-pass-additional-vm_flags-to-__vmalloc_node_range.patch kernel-add-support-for-init_array-constructors.patch module-fix-types-of-device-tables-aliases.patch kasan-enable-instrumentation-of-global-variables.patch -- To unsubscribe from this list: send the line "unsubscribe mm-commits" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html