On Wed, Oct 2, 2024 at 4:08 AM Christoph Lameter via B4 Relay <devnull+cl.gentwo.org@xxxxxxxxxx> wrote: > > From: Christoph Lameter <cl@xxxxxxxxxx> > > The old SLAB allocator used to support memory policies on a per > allocation bases. In SLUB the memory policies are applied on a > per page frame / folio bases. Doing so avoids having to check memory > policies in critical code paths for kmalloc and friends. > > This worked on general well on Intel/AMD/PowerPC because the > interconnect technology is mature and can minimize the latencies > through intelligent caching even if a small object is not > placed optimally. > > However, on ARM we have an emergence of new NUMA interconnect > technology based more on embedded devices. Caching of remote content > can currently be ineffective using the standard building blocks / mesh > available on that platform. Such architectures benefit if each slab > object is individually placed according to memory policies > and other restrictions. > > This patch adds another kernel parameter > > slab_strict_numa > > If that is set then a static branch is activated that will cause > the hotpaths of the allocator to evaluate the current memory > allocation policy. Each object will be properly placed by > paying the price of extra processing and SLUB will no longer > defer to the page allocator to apply memory policies at the > folio level. > > This patch improves performance of memcached running > on Ampere Altra 2P system (ARM Neoverse N1 processor) > by 3.6% due to accurate placement of small kernel objects. > > Tested-by: Huang Shijie <shijie@xxxxxxxxxxxxxxxxxxxxxx> > Signed-off-by: Christoph Lameter (Ampere) <cl@xxxxxxxxxx> > --- > Changes in v3: > - Make the static key a static in slub.c > - Use pr_warn / pr_info instead of printk > - Link to v2: https://lore.kernel.org/r/20240906-strict_numa-v2-1-f104e6de6d1e@xxxxxxxxxx > > Changes in v2: > - Fix various issues > - Testing > --- > mm/slub.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 42 insertions(+) > > diff --git a/mm/slub.c b/mm/slub.c > index 21f71cb6cc06..7ae94f79740d 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -218,6 +218,10 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); > #endif > #endif /* CONFIG_SLUB_DEBUG */ > > +#ifdef CONFIG_NUMA > +static DEFINE_STATIC_KEY_FALSE(strict_numa); > +#endif > + > /* Structure holding parameters for get_partial() call chain */ > struct partial_context { > gfp_t flags; > @@ -3957,6 +3961,28 @@ static __always_inline void *__slab_alloc_node(struct kmem_cache *s, > object = c->freelist; > slab = c->slab; > > +#ifdef CONFIG_NUMA > + if (static_branch_unlikely(&strict_numa) && > + node == NUMA_NO_NODE) { > + > + struct mempolicy *mpol = current->mempolicy; > + > + if (mpol) { > + /* > + * Special BIND rule support. If existing slab > + * is in permitted set then do not redirect > + * to a particular node. > + * Otherwise we apply the memory policy to get > + * the node we need to allocate on. > + */ > + if (mpol->mode != MPOL_BIND || !slab || > + !node_isset(slab_nid(slab), mpol->nodes)) > + > + node = mempolicy_slab_node(); > + } Is it intentional to allow the local node only (via mempolicy_slab_node()) in interrupt contexts? > + } > +#endif > + > if (!USE_LOCKLESS_FAST_PATH() || > unlikely(!object || !slab || !node_match(slab, node))) { > object = __slab_alloc(s, gfpflags, node, addr, c, orig_size); > @@ -5601,6 +5627,22 @@ static int __init setup_slub_min_objects(char *str) > __setup("slab_min_objects=", setup_slub_min_objects); > __setup_param("slub_min_objects=", slub_min_objects, setup_slub_min_objects, 0); > > +#ifdef CONFIG_NUMA > +static int __init setup_slab_strict_numa(char *str) > +{ > + if (nr_node_ids > 1) { > + static_branch_enable(&strict_numa); > + pr_info("SLUB: Strict NUMA enabled.\n"); > + } else > + pr_warn("slab_strict_numa parameter set on non NUMA system.\n"); nit: this statement should be enclosed within braces per coding style guideline. Otherwise everything looks good to me (including the document amended). Best, Hyeonggon
