The patch titled
Subject: mm/vmscan.c: iterate only over charged shrinkers during memcg shrink_slab()
has been added to the -mm tree. Its filename is
mm-iterate-only-over-charged-shrinkers-during-memcg-shrink_slab.patch
This patch should soon appear at
http://ozlabs.org/~akpm/mmots/broken-out/mm-iterate-only-over-charged-shrinkers-during-memcg-shrink_slab.patch
and later at
http://ozlabs.org/~akpm/mmotm/broken-out/mm-iterate-only-over-charged-shrinkers-during-memcg-shrink_slab.patch
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
The -mm tree is included into linux-next and is updated
there every 3-4 working days
------------------------------------------------------
From: Kirill Tkhai <ktkhai@xxxxxxxxxxxxx>
Subject: mm/vmscan.c: iterate only over charged shrinkers during memcg shrink_slab()
Using the preparations made in previous patches, in case of memcg shrink,
we may avoid shrinkers, which are not set in memcg's shrinkers bitmap. To
do that, we separate iterations over memcg-aware and !memcg-aware
shrinkers, and memcg-aware shrinkers are chosen via for_each_set_bit()
from the bitmap. In case of big nodes, having many isolated environments,
this gives significant performance growth. See next patches for the
details.
Note, that the patch does not respect to empty memcg shrinkers, since we
never clear the bitmap bits after we set it once. Their shrinkers will be
called again, with no shrinked objects as result. This functionality is
provided by next patches.
Link: http://lkml.kernel.org/r/153063066653.1818.976035462801487910.stgit@localhost.localdomain
Signed-off-by: Kirill Tkhai <ktkhai@xxxxxxxxxxxxx>
Acked-by: Vladimir Davydov <vdavydov.dev@xxxxxxxxx>
Tested-by: Shakeel Butt <shakeelb@xxxxxxxxxx>
Cc: Al Viro <viro@xxxxxxxxxxxxxxxxxx>
Cc: Andrey Ryabinin <aryabinin@xxxxxxxxxxxxx>
Cc: Chris Wilson <chris@xxxxxxxxxxxxxxxxxx>
Cc: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
Cc: Guenter Roeck <linux@xxxxxxxxxxxx>
Cc: "Huang, Ying" <ying.huang@xxxxxxxxx>
Cc: Johannes Weiner <hannes@xxxxxxxxxxx>
Cc: Josef Bacik <jbacik@xxxxxx>
Cc: Li RongQing <lirongqing@xxxxxxxxx>
Cc: Matthew Wilcox <willy@xxxxxxxxxxxxx>
Cc: Matthias Kaehlcke <mka@xxxxxxxxxxxx>
Cc: Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx>
Cc: Michal Hocko <mhocko@xxxxxxxxxx>
Cc: Minchan Kim <minchan@xxxxxxxxxx>
Cc: Philippe Ombredanne <pombredanne@xxxxxxxx>
Cc: Roman Gushchin <guro@xxxxxx>
Cc: Sahitya Tummala <stummala@xxxxxxxxxxxxxx>
Cc: Stephen Rothwell <sfr@xxxxxxxxxxxxxxxx>
Cc: Tetsuo Handa <penguin-kernel@xxxxxxxxxxxxxxxxxxx>
Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Cc: Waiman Long <longman@xxxxxxxxxx>
Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
---
mm/vmscan.c | 87 ++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 78 insertions(+), 9 deletions(-)
diff -puN mm/vmscan.c~mm-iterate-only-over-charged-shrinkers-during-memcg-shrink_slab mm/vmscan.c
--- a/mm/vmscan.c~mm-iterate-only-over-charged-shrinkers-during-memcg-shrink_slab
+++ a/mm/vmscan.c
@@ -367,6 +367,20 @@ int prealloc_shrinker(struct shrinker *s
goto free_deferred;
}
+ /*
+ * There is a window between prealloc_shrinker()
+ * and register_shrinker_prepared(). We don't want
+ * to clear bit of a shrinker in such the state
+ * in shrink_slab_memcg(), since this will impose
+ * restrictions on a code registering a shrinker
+ * (they would have to guarantee, their LRU lists
+ * are empty till shrinker is completely registered).
+ * So, we differ the situation, when 1)a shrinker
+ * is semi-registered (id is assigned, but it has
+ * not yet linked to shrinker_list) and 2)shrinker
+ * is not registered (id is not assigned).
+ */
+ INIT_LIST_HEAD(&shrinker->list);
return 0;
free_deferred:
@@ -541,6 +555,67 @@ static unsigned long do_shrink_slab(stru
return freed;
}
+#ifdef CONFIG_MEMCG_KMEM
+static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
+ struct mem_cgroup *memcg, int priority)
+{
+ struct memcg_shrinker_map *map;
+ unsigned long freed = 0;
+ int ret, i;
+
+ if (!memcg_kmem_enabled() || !mem_cgroup_online(memcg))
+ return 0;
+
+ if (!down_read_trylock(&shrinker_rwsem))
+ return 0;
+
+ /*
+ * 1) Caller passes only alive memcg, so map can't be NULL.
+ * 2) shrinker_rwsem protects from maps expanding.
+ */
+ map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map,
+ true);
+ BUG_ON(!map);
+
+ for_each_set_bit(i, map->map, shrinker_nr_max) {
+ struct shrink_control sc = {
+ .gfp_mask = gfp_mask,
+ .nid = nid,
+ .memcg = memcg,
+ };
+ struct shrinker *shrinker;
+
+ shrinker = idr_find(&shrinker_idr, i);
+ if (unlikely(!shrinker)) {
+ clear_bit(i, map->map);
+ continue;
+ }
+ BUG_ON(!(shrinker->flags & SHRINKER_MEMCG_AWARE));
+
+ /* See comment in prealloc_shrinker() */
+ if (unlikely(list_empty(&shrinker->list)))
+ continue;
+
+ ret = do_shrink_slab(&sc, shrinker, priority);
+ freed += ret;
+
+ if (rwsem_is_contended(&shrinker_rwsem)) {
+ freed = freed ? : 1;
+ break;
+ }
+ }
+
+ up_read(&shrinker_rwsem);
+ return freed;
+}
+#else /* CONFIG_MEMCG_KMEM */
+static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
+ struct mem_cgroup *memcg, int priority)
+{
+ return 0;
+}
+#endif /* CONFIG_MEMCG_KMEM */
+
/**
* shrink_slab - shrink slab caches
* @gfp_mask: allocation context
@@ -570,8 +645,8 @@ static unsigned long shrink_slab(gfp_t g
struct shrinker *shrinker;
unsigned long freed = 0;
- if (memcg && (!memcg_kmem_enabled() || !mem_cgroup_online(memcg)))
- return 0;
+ if (memcg && !mem_cgroup_is_root(memcg))
+ return shrink_slab_memcg(gfp_mask, nid, memcg, priority);
if (!down_read_trylock(&shrinker_rwsem))
goto out;
@@ -583,13 +658,7 @@ static unsigned long shrink_slab(gfp_t g
.memcg = memcg,
};
- /*
- * If kernel memory accounting is disabled, we ignore
- * SHRINKER_MEMCG_AWARE flag and call all shrinkers
- * passing NULL for memcg.
- */
- if (memcg_kmem_enabled() &&
- !!memcg != !!(shrinker->flags & SHRINKER_MEMCG_AWARE))
+ if (!!memcg != !!(shrinker->flags & SHRINKER_MEMCG_AWARE))
continue;
if (!(shrinker->flags & SHRINKER_NUMA_AWARE))
_
Patches currently in -mm which might be from ktkhai@xxxxxxxxxxxxx are
memcg-remove-memcg_cgroup-id-from-idr-on-mem_cgroup_css_alloc-failure.patch
list_lru-combine-code-under-the-same-define.patch
mm-introduce-config_memcg_kmem-as-combination-of-config_memcg-config_slob.patch
mm-assign-id-to-every-memcg-aware-shrinker.patch
memcg-move-up-for_each_mem_cgroup-_tree-defines.patch
mm-assign-memcg-aware-shrinkers-bitmap-to-memcg.patch
mm-refactoring-in-workingset_init.patch
fs-refactoring-in-alloc_super.patch
fs-propagate-shrinker-id-to-list_lru.patch
list_lru-add-memcg-argument-to-list_lru_from_kmem.patch
list_lru-pass-dst_memcg-argument-to-memcg_drain_list_lru_node.patch
list_lru-pass-lru-argument-to-memcg_drain_list_lru_node.patch
mm-export-mem_cgroup_is_root.patch
mm-set-bit-in-memcg-shrinker-bitmap-on-first-list_lru-item-apearance.patch
mm-iterate-only-over-charged-shrinkers-during-memcg-shrink_slab.patch
mm-add-shrink_empty-shrinker-methods-return-value.patch
mm-clear-shrinker-bit-if-there-are-no-objects-related-to-memcg.patch
--
To unsubscribe from this list: send the line "unsubscribe mm-commits" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
