On Tue, Feb 22, 2022 at 12:22:02PM +0300, Vasily Averin wrote:
> Dear all,
>
> I've found that /proc/slabinfo shows inadequate numbers of in-use slab objects.
> it assumes that all objects stored in cpu caches are always 100% in use.
>
> for example:
> slabinfo shows that all 20 objects are in use.
>
> [root@fc34-vvs linux]# uname -a
> Linux fc34-vvs.sw.ru 5.17.0-rc3+ #42 SMP PREEMPT Mon Feb 21 20:14:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> [root@fc34-vvs linux]# cat /proc/slabinfo
> slabinfo - version: 2.1
> # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
> ...
> kmalloc-cg-8k 20 20 8192 4 8 : tunables 0 0 0 : slabdata 5 5 0
>
> At the same time crash said that only 2 objects are in use.
>
> crash> kmem -s kmalloc-cg-8k
> CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
> ffff8f4840043b00 8192 2 20 5 32k kmalloc-cg-8k
>
> And this looks like true, see kmem -S output below.
>
> Is it a bug or perhaps a well-known feature that I missed?
>
This is not a bug..
TL;DR: Some of slabs are locked by CPU and SLUB does not accurately account them.
(I guess its purpose is to reduce cacheline usages in fastpath?)
There are 3 status of slab. A slab can be:
1) a cpu slab or 2) cpu partial slabs:
They're locked (frozen) by cpu to avoid per-node
locking overhead.
3) node partial slabs:
They exist in node's partial slab list. Any CPU can take them
using spinlock. Usually SLUB takes slab from node
partial slabs when there is no cpu slab and cpu partial
slabs.
Only free objects in node partial slabs are calculated as free objects.
> Numbers are counted in mm/slub.c, see below,
> but count_partial() doe not includes free objects of cpu caches
>
> Moreover adequate statistic is not showed in any other interfaces too
> /sys/kerenl/slab/ read cpu slab caches but does not output these numbers.
>
> Thank you,
> Vasily Averin
>
> #ifdef CONFIG_SLUB_DEBUG
> void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo)
> {
> unsigned long nr_slabs = 0;
> unsigned long nr_objs = 0;
> unsigned long nr_free = 0;
> int node;
> struct kmem_cache_node *n;
>
> for_each_kmem_cache_node(s, node, n) {
> nr_slabs += node_nr_slabs(n);
> nr_objs += node_nr_objs(n);
> nr_free += count_partial(n, count_free);
> }
>
> sinfo->active_objs = nr_objs - nr_free;
This code assumes all objects in frozen slabs (cpu slab or cpu
partial slabs) are not free.
With current implementation of SLUB, there is no easy way to accurately
account objects in cpu slab and cpu partial slabs' objects because SLUB
sets slab->inuse = slab->objects when taking slab from node partial slabs.
They are accounted again only when going back to node partial slabs.
(deactivated).
Thanks,
Hyeonggon
> sinfo->num_objs = nr_objs;
>
>
>
> crash> kmem -S kmalloc-cg-8k
> CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
> ffff8f4840043b00 8192 2 20 5 32k kmalloc-cg-8k
> CPU 0 KMEM_CACHE_CPU:
> ffff8f4b58236360
> CPU 0 SLAB:
> (empty)
> CPU 0 PARTIAL:
> (empty)
> CPU 1 KMEM_CACHE_CPU:
> ffff8f4b58276360
> CPU 1 SLAB:
> SLAB MEMORY NODE TOTAL ALLOCATED FREE
> ffffed3f842af400 ffff8f484abd0000 0 4 1 3
> FREE / [ALLOCATED]
> ffff8f484abd0000 (cpu 1 cache)
> ffff8f484abd2000 (cpu 1 cache)
> ffff8f484abd4000 (cpu 1 cache)
> [ffff8f484abd6000]
> CPU 1 PARTIAL:
> (empty)
> CPU 2 KMEM_CACHE_CPU:
> ffff8f4b582b6360
> CPU 2 SLAB:
> (empty)
> CPU 2 PARTIAL:
> (empty)
> CPU 3 KMEM_CACHE_CPU:
> ffff8f4b582f6360
> CPU 3 SLAB:
> SLAB MEMORY NODE TOTAL ALLOCATED FREE
> ffffed3f842ce600 ffff8f484b398000 0 4 0 4
> FREE / [ALLOCATED]
> ffff8f484b398000 (cpu 3 cache)
> ffff8f484b39a000 (cpu 3 cache)
> ffff8f484b39c000 (cpu 3 cache)
> ffff8f484b39e000 (cpu 3 cache)
> CPU 3 PARTIAL:
> (empty)
> CPU 4 KMEM_CACHE_CPU:
> ffff8f4b58336360
> CPU 4 SLAB:
> SLAB MEMORY NODE TOTAL ALLOCATED FREE
> ffffed3f8418c200 ffff8f4846308000 0 4 0 4
> FREE / [ALLOCATED]
> ffff8f4846308000 (cpu 4 cache)
> ffff8f484630a000 (cpu 4 cache)
> ffff8f484630c000 (cpu 4 cache)
> ffff8f484630e000 (cpu 4 cache)
> CPU 4 PARTIAL:
> (empty)
> CPU 5 KMEM_CACHE_CPU:
> ffff8f4b58376360
> CPU 5 SLAB:
> (empty)
> CPU 5 PARTIAL:
> (empty)
> CPU 6 KMEM_CACHE_CPU:
> ffff8f4b583b6360
> CPU 6 SLAB:
> SLAB MEMORY NODE TOTAL ALLOCATED FREE
> ffffed3f8412d000 ffff8f4844b40000 0 4 0 4
> FREE / [ALLOCATED]
> ffff8f4844b40000 (cpu 6 cache)
> ffff8f4844b42000 (cpu 6 cache)
> ffff8f4844b44000 (cpu 6 cache)
> ffff8f4844b46000 (cpu 6 cache)
> CPU 6 PARTIAL:
> (empty)
> CPU 7 KMEM_CACHE_CPU:
> ffff8f4b583f6360
> CPU 7 SLAB:
> SLAB MEMORY NODE TOTAL ALLOCATED FREE
> ffffed3f84124000 ffff8f4844900000 0 4 1 3
> FREE / [ALLOCATED]
> ffff8f4844900000 (cpu 7 cache)
> ffff8f4844902000 (cpu 7 cache)
> [ffff8f4844904000]
> ffff8f4844906000 (cpu 7 cache)
> CPU 7 PARTIAL:
> (empty)
> KMEM_CACHE_NODE NODE SLABS PARTIAL PER-CPU
> ffff8f48400416c0 0 5 0 5
> NODE 0 PARTIAL:
> (empty)
> NODE 0 FULL:
> (not tracked)
>
>
>
