Hm looks like I forgot To: Andrew some kind of feedback from fellow slab maintainers also wouldn't hurt :) Thanks. On 10/12/21 15:46, Vlastimil Babka wrote: > With CONFIG_SLUB_CPU_PARTIAL enabled, SLUB keeps a percpu list of partial > slabs that can be promoted to cpu slab when the previous one is depleted, > without accessing the shared partial list. A slab can be added to this list > by 1) refill of an empty list from get_partial_node() - once we really have to > access the shared partial list, we acquire multiple slabs to amortize the cost > of locking, and 2) first free to a previously full slab - instead of putting > the slab on a shared partial list, we can more cheaply freeze it and put it on > the per-cpu list. > > To control how large a percpu partial list can grow for a kmem cache, > set_cpu_partial() calculates a target number of free objects on each cpu's > percpu partial list, and this can be also set by the sysfs file cpu_partial. > > However, the tracking of actual number of objects is imprecise, in order to > limit overhead from cpu X freeing an objects to a slab on percpu partial > list of cpu Y. Basically, the percpu partial slabs form a single linked list, > and when we add a new slab to the list with current head "oldpage", we set in > the struct page of the slab we're adding: > > page->pages = oldpage->pages + 1; // this is precise > page->pobjects = oldpage->pobjects + (page->objects - page->inuse); > page->next = oldpage; > > Thus the real number of free objects in the slab (objects - inuse) is only > determined at the moment of adding the slab to the percpu partial list, and > further freeing doesn't update the pobjects counter nor propagate it to the > current list head. As Jann reports [1], this can easily lead to large > inaccuracies, where the target number of objects (up to 30 by default) can > translate to the same number of (empty) slab pages on the list. In case 2) > above, we put a slab with 1 free object on the list, thus only increase > page->pobjects by 1, even if there are subsequent frees on the same slab. Jann > has noticed this in practice and so did we [2] when investigating significant > increase of kmemcg usage after switching from SLAB to SLUB. > > While this is no longer a problem in kmemcg context thanks to the accounting > rewrite in 5.9, the memory waste is still not ideal and it's questionable > whether it makes sense to perform free object count based control when object > counts can easily become so much inaccurate. So this patch converts the > accounting to be based on number of pages only (which is precise) and removes > the page->pobjects field completely. This is also ultimately simpler. > > To retain the existing set_cpu_partial() heuristic, first calculate the target > number of objects as previously, but then convert it to target number of pages > by assuming the pages will be half-filled on average. This assumption might > obviously also be inaccurate in practice, but cannot degrade to actual number of > pages being equal to the target number of objects. > > We could also skip the intermediate step with target number of objects and > rewrite the heuristic in terms of pages. However we still have the sysfs file > cpu_partial which uses number of objects and could break existing users if it > suddenly becomes number of pages, so this patch doesn't do that. > > In practice, after this patch the heuristics limit the size of percpu partial > list up to 2 pages. In case of a reported regression (which would mean some > workload has benefited from the previous imprecise object based counting), we > can tune the heuristics to get a better compromise within the new scheme, while > still avoid the unexpectedly long percpu partial lists. > > [1] https://lore.kernel.org/linux-mm/CAG48ez2Qx5K1Cab-m8BdSibp6wLTip6ro4=-umR7BLsEgjEYzA@xxxxxxxxxxxxxx/ > [2] https://lore.kernel.org/all/2f0f46e8-2535-410a-1859-e9cfa4e57c18@xxxxxxx/ > > ========== > Evaluation > ========== > > Mel was kind enough to run v1 through mmtests machinery for netperf (localhost) > and hackbench and, for most significant results see below. So there are some > apparent regressions, especially with hackbench, which I think ultimately boils > down to having shorter percpu partial lists on average and some benchmarks > benefiting from longer ones. Monitoring slab usage also indicated less memory > usage by slab. Based on that, the following patch will bump the defaults to > allow longer percpu partial lists than after this patch. > > However the goal is certainly not such that we would limit the percpu partial > lists to 30 pages just because previously a specific alloc/free pattern could > lead to the limit of 30 objects translate to a limit to 30 pages - that would > make little sense. This is a correctness patch, and if a workload benefits from > larger lists, the sysfs tuning knobs are still there to allow that. > > Netperf > > 2-socket Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz (20 cores, 40 threads > per socket), 384GB RAM > TCP-RR: > hmean before 127045.79 after 121092.94 (-4.69%, worse) > stddev before 2634.37 after 1254.08 > UDP-RR: > hmean before 166985.45 after 160668.94 ( -3.78%, worse) > stddev before 4059.69 after 1943.63 > > 2-socket Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz (20 cores, 40 threads > per socket), 512GB RAM > TCP-RR: > hmean before 84173.25 after 76914.72 ( -8.62%, worse) > UDP-RR: > hmean before 93571.12 after 96428.69 ( 3.05%, better) > stddev before 23118.54 after 16828.14 > > 2-socket Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz (12 cores, 24 threads > per socket), 64GB RAM > TCP-RR: > hmean before 49984.92 after 48922.27 ( -2.13%, worse) > stddev before 6248.15 after 4740.51 > UDP-RR: > hmean before 61854.31 after 68761.81 ( 11.17%, better) > stddev before 4093.54 after 5898.91 > > other machines - within 2% > > Hackbench > > (results before and after the patch, negative % means worse) > > 2-socket AMD EPYC 7713 (64 cores, 128 threads per core), 256GB RAM > hackbench-process-sockets > Amean 1 0.5380 0.5583 ( -3.78%) > Amean 4 0.7510 0.8150 ( -8.52%) > Amean 7 0.7930 0.9533 ( -20.22%) > Amean 12 0.7853 1.1313 ( -44.06%) > Amean 21 1.1520 1.4993 ( -30.15%) > Amean 30 1.6223 1.9237 ( -18.57%) > Amean 48 2.6767 2.9903 ( -11.72%) > Amean 79 4.0257 5.1150 ( -27.06%) > Amean 110 5.5193 7.4720 ( -35.38%) > Amean 141 7.2207 9.9840 ( -38.27%) > Amean 172 8.4770 12.1963 ( -43.88%) > Amean 203 9.6473 14.3137 ( -48.37%) > Amean 234 11.3960 18.7917 ( -64.90%) > Amean 265 13.9627 22.4607 ( -60.86%) > Amean 296 14.9163 26.0483 ( -74.63%) > > hackbench-thread-sockets > Amean 1 0.5597 0.5877 ( -5.00%) > Amean 4 0.7913 0.8960 ( -13.23%) > Amean 7 0.8190 1.0017 ( -22.30%) > Amean 12 0.9560 1.1727 ( -22.66%) > Amean 21 1.7587 1.5660 ( 10.96%) > Amean 30 2.4477 1.9807 ( 19.08%) > Amean 48 3.4573 3.0630 ( 11.41%) > Amean 79 4.7903 5.1733 ( -8.00%) > Amean 110 6.1370 7.4220 ( -20.94%) > Amean 141 7.5777 9.2617 ( -22.22%) > Amean 172 9.2280 11.0907 ( -20.18%) > Amean 203 10.2793 13.3470 ( -29.84%) > Amean 234 11.2410 17.1070 ( -52.18%) > Amean 265 12.5970 23.3323 ( -85.22%) > Amean 296 17.1540 24.2857 ( -41.57%) > > 2-socket Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz (20 cores, 40 threads > per socket), 384GB RAM > hackbench-process-sockets > Amean 1 0.5760 0.4793 ( 16.78%) > Amean 4 0.9430 0.9707 ( -2.93%) > Amean 7 1.5517 1.8843 ( -21.44%) > Amean 12 2.4903 2.7267 ( -9.49%) > Amean 21 3.9560 4.2877 ( -8.38%) > Amean 30 5.4613 5.8343 ( -6.83%) > Amean 48 8.5337 9.2937 ( -8.91%) > Amean 79 14.0670 15.2630 ( -8.50%) > Amean 110 19.2253 21.2467 ( -10.51%) > Amean 141 23.7557 25.8550 ( -8.84%) > Amean 172 28.4407 29.7603 ( -4.64%) > Amean 203 33.3407 33.9927 ( -1.96%) > Amean 234 38.3633 39.1150 ( -1.96%) > Amean 265 43.4420 43.8470 ( -0.93%) > Amean 296 48.3680 48.9300 ( -1.16%) > > hackbench-thread-sockets > Amean 1 0.6080 0.6493 ( -6.80%) > Amean 4 1.0000 1.0513 ( -5.13%) > Amean 7 1.6607 2.0260 ( -22.00%) > Amean 12 2.7637 2.9273 ( -5.92%) > Amean 21 5.0613 4.5153 ( 10.79%) > Amean 30 6.3340 6.1140 ( 3.47%) > Amean 48 9.0567 9.5577 ( -5.53%) > Amean 79 14.5657 15.7983 ( -8.46%) > Amean 110 19.6213 21.6333 ( -10.25%) > Amean 141 24.1563 26.2697 ( -8.75%) > Amean 172 28.9687 30.2187 ( -4.32%) > Amean 203 33.9763 34.6970 ( -2.12%) > Amean 234 38.8647 39.3207 ( -1.17%) > Amean 265 44.0813 44.1507 ( -0.16%) > Amean 296 49.2040 49.4330 ( -0.47%) > > 2-socket Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz (20 cores, 40 threads > per socket), 512GB RAM > hackbench-process-sockets > Amean 1 0.5027 0.5017 ( 0.20%) > Amean 4 1.1053 1.2033 ( -8.87%) > Amean 7 1.8760 2.1820 ( -16.31%) > Amean 12 2.9053 3.1810 ( -9.49%) > Amean 21 4.6777 4.9920 ( -6.72%) > Amean 30 6.5180 6.7827 ( -4.06%) > Amean 48 10.0710 10.5227 ( -4.48%) > Amean 79 16.4250 17.5053 ( -6.58%) > Amean 110 22.6203 24.4617 ( -8.14%) > Amean 141 28.0967 31.0363 ( -10.46%) > Amean 172 34.4030 36.9233 ( -7.33%) > Amean 203 40.5933 43.0850 ( -6.14%) > Amean 234 46.6477 48.7220 ( -4.45%) > Amean 265 53.0530 53.9597 ( -1.71%) > Amean 296 59.2760 59.9213 ( -1.09%) > > hackbench-thread-sockets > Amean 1 0.5363 0.5330 ( 0.62%) > Amean 4 1.1647 1.2157 ( -4.38%) > Amean 7 1.9237 2.2833 ( -18.70%) > Amean 12 2.9943 3.3110 ( -10.58%) > Amean 21 4.9987 5.1880 ( -3.79%) > Amean 30 6.7583 7.0043 ( -3.64%) > Amean 48 10.4547 10.8353 ( -3.64%) > Amean 79 16.6707 17.6790 ( -6.05%) > Amean 110 22.8207 24.4403 ( -7.10%) > Amean 141 28.7090 31.0533 ( -8.17%) > Amean 172 34.9387 36.8260 ( -5.40%) > Amean 203 41.1567 43.0450 ( -4.59%) > Amean 234 47.3790 48.5307 ( -2.43%) > Amean 265 53.9543 54.6987 ( -1.38%) > Amean 296 60.0820 60.2163 ( -0.22%) > > 1-socket Intel(R) Xeon(R) CPU E3-1240 v5 @ 3.50GHz (4 cores, 8 threads), > 32 GB RAM > hackbench-process-sockets > Amean 1 1.4760 1.5773 ( -6.87%) > Amean 3 3.9370 4.0910 ( -3.91%) > Amean 5 6.6797 6.9357 ( -3.83%) > Amean 7 9.3367 9.7150 ( -4.05%) > Amean 12 15.7627 16.1400 ( -2.39%) > Amean 18 23.5360 23.6890 ( -0.65%) > Amean 24 31.0663 31.3137 ( -0.80%) > Amean 30 38.7283 39.0037 ( -0.71%) > Amean 32 41.3417 41.6097 ( -0.65%) > > hackbench-thread-sockets > Amean 1 1.5250 1.6043 ( -5.20%) > Amean 3 4.0897 4.2603 ( -4.17%) > Amean 5 6.7760 7.0933 ( -4.68%) > Amean 7 9.4817 9.9157 ( -4.58%) > Amean 12 15.9610 16.3937 ( -2.71%) > Amean 18 23.9543 24.3417 ( -1.62%) > Amean 24 31.4400 31.7217 ( -0.90%) > Amean 30 39.2457 39.5467 ( -0.77%) > Amean 32 41.8267 42.1230 ( -0.71%) > > 2-socket Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz (12 cores, 24 threads > per socket), 64GB RAM > hackbench-process-sockets > Amean 1 1.0347 1.0880 ( -5.15%) > Amean 4 1.7267 1.8527 ( -7.30%) > Amean 7 2.6707 2.8110 ( -5.25%) > Amean 12 4.1617 4.3383 ( -4.25%) > Amean 21 7.0070 7.2600 ( -3.61%) > Amean 30 9.9187 10.2397 ( -3.24%) > Amean 48 15.6710 16.3923 ( -4.60%) > Amean 79 24.7743 26.1247 ( -5.45%) > Amean 110 34.3000 35.9307 ( -4.75%) > Amean 141 44.2043 44.8010 ( -1.35%) > Amean 172 54.2430 54.7260 ( -0.89%) > Amean 192 60.6557 60.9777 ( -0.53%) > > hackbench-thread-sockets > Amean 1 1.0610 1.1353 ( -7.01%) > Amean 4 1.7543 1.9140 ( -9.10%) > Amean 7 2.7840 2.9573 ( -6.23%) > Amean 12 4.3813 4.4937 ( -2.56%) > Amean 21 7.3460 7.5350 ( -2.57%) > Amean 30 10.2313 10.5190 ( -2.81%) > Amean 48 15.9700 16.5940 ( -3.91%) > Amean 79 25.3973 26.6637 ( -4.99%) > Amean 110 35.1087 36.4797 ( -3.91%) > Amean 141 45.8220 46.3053 ( -1.05%) > Amean 172 55.4917 55.7320 ( -0.43%) > Amean 192 62.7490 62.5410 ( 0.33%) > > Reported-by: Jann Horn <jannh@xxxxxxxxxx> > Signed-off-by: Vlastimil Babka <vbabka@xxxxxxx> > --- > Changes in v2: > - added evaluation results to changelog > - added patch 2 bumping the defaults > include/linux/mm_types.h | 2 - > include/linux/slub_def.h | 13 +----- > mm/slub.c | 89 ++++++++++++++++++++++++++-------------- > 3 files changed, 61 insertions(+), 43 deletions(-) > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > index 7f8ee09c711f..68ffa064b7a8 100644 > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -124,10 +124,8 @@ struct page { > struct page *next; > #ifdef CONFIG_64BIT > int pages; /* Nr of pages left */ > - int pobjects; /* Approximate count */ > #else > short int pages; > - short int pobjects; > #endif > }; > }; > diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h > index 85499f0586b0..0fa751b946fa 100644 > --- a/include/linux/slub_def.h > +++ b/include/linux/slub_def.h > @@ -99,6 +99,8 @@ struct kmem_cache { > #ifdef CONFIG_SLUB_CPU_PARTIAL > /* Number of per cpu partial objects to keep around */ > unsigned int cpu_partial; > + /* Number of per cpu partial pages to keep around */ > + unsigned int cpu_partial_pages; > #endif > struct kmem_cache_order_objects oo; > > @@ -141,17 +143,6 @@ struct kmem_cache { > struct kmem_cache_node *node[MAX_NUMNODES]; > }; > > -#ifdef CONFIG_SLUB_CPU_PARTIAL > -#define slub_cpu_partial(s) ((s)->cpu_partial) > -#define slub_set_cpu_partial(s, n) \ > -({ \ > - slub_cpu_partial(s) = (n); \ > -}) > -#else > -#define slub_cpu_partial(s) (0) > -#define slub_set_cpu_partial(s, n) > -#endif /* CONFIG_SLUB_CPU_PARTIAL */ > - > #ifdef CONFIG_SYSFS > #define SLAB_SUPPORTS_SYSFS > void sysfs_slab_unlink(struct kmem_cache *); > diff --git a/mm/slub.c b/mm/slub.c > index 3d2025f7163b..3757f31c5d97 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -414,6 +414,29 @@ static inline unsigned int oo_objects(struct kmem_cache_order_objects x) > return x.x & OO_MASK; > } > > +#ifdef CONFIG_SLUB_CPU_PARTIAL > +static void slub_set_cpu_partial(struct kmem_cache *s, unsigned int nr_objects) > +{ > + unsigned int nr_pages; > + > + s->cpu_partial = nr_objects; > + > + /* > + * We take the number of objects but actually limit the number of > + * pages on the per cpu partial list, in order to limit excessive > + * growth of the list. For simplicity we assume that the pages will > + * be half-full. > + */ > + nr_pages = DIV_ROUND_UP(nr_objects * 2, oo_objects(s->oo)); > + s->cpu_partial_pages = nr_pages; > +} > +#else > +static inline void > +slub_set_cpu_partial(struct kmem_cache *s, unsigned int nr_objects) > +{ > +} > +#endif /* CONFIG_SLUB_CPU_PARTIAL */ > + > /* > * Per slab locking using the pagelock > */ > @@ -2045,7 +2068,7 @@ static inline void remove_partial(struct kmem_cache_node *n, > */ > static inline void *acquire_slab(struct kmem_cache *s, > struct kmem_cache_node *n, struct page *page, > - int mode, int *objects) > + int mode) > { > void *freelist; > unsigned long counters; > @@ -2061,7 +2084,6 @@ static inline void *acquire_slab(struct kmem_cache *s, > freelist = page->freelist; > counters = page->counters; > new.counters = counters; > - *objects = new.objects - new.inuse; > if (mode) { > new.inuse = page->objects; > new.freelist = NULL; > @@ -2099,9 +2121,8 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, > { > struct page *page, *page2; > void *object = NULL; > - unsigned int available = 0; > unsigned long flags; > - int objects; > + unsigned int partial_pages = 0; > > /* > * Racy check. If we mistakenly see no partial slabs then we > @@ -2119,11 +2140,10 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, > if (!pfmemalloc_match(page, gfpflags)) > continue; > > - t = acquire_slab(s, n, page, object == NULL, &objects); > + t = acquire_slab(s, n, page, object == NULL); > if (!t) > break; > > - available += objects; > if (!object) { > *ret_page = page; > stat(s, ALLOC_FROM_PARTIAL); > @@ -2131,10 +2151,15 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, > } else { > put_cpu_partial(s, page, 0); > stat(s, CPU_PARTIAL_NODE); > + partial_pages++; > } > +#ifdef CONFIG_SLUB_CPU_PARTIAL > if (!kmem_cache_has_cpu_partial(s) > - || available > slub_cpu_partial(s) / 2) > + || partial_pages > s->cpu_partial_pages / 2) > break; > +#else > + break; > +#endif > > } > spin_unlock_irqrestore(&n->list_lock, flags); > @@ -2539,14 +2564,13 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) > struct page *page_to_unfreeze = NULL; > unsigned long flags; > int pages = 0; > - int pobjects = 0; > > local_lock_irqsave(&s->cpu_slab->lock, flags); > > oldpage = this_cpu_read(s->cpu_slab->partial); > > if (oldpage) { > - if (drain && oldpage->pobjects > slub_cpu_partial(s)) { > + if (drain && oldpage->pages >= s->cpu_partial_pages) { > /* > * Partial array is full. Move the existing set to the > * per node partial list. Postpone the actual unfreezing > @@ -2555,16 +2579,13 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) > page_to_unfreeze = oldpage; > oldpage = NULL; > } else { > - pobjects = oldpage->pobjects; > pages = oldpage->pages; > } > } > > pages++; > - pobjects += page->objects - page->inuse; > > page->pages = pages; > - page->pobjects = pobjects; > page->next = oldpage; > > this_cpu_write(s->cpu_slab->partial, page); > @@ -3980,6 +4001,8 @@ static void set_min_partial(struct kmem_cache *s, unsigned long min) > static void set_cpu_partial(struct kmem_cache *s) > { > #ifdef CONFIG_SLUB_CPU_PARTIAL > + unsigned int nr_objects; > + > /* > * cpu_partial determined the maximum number of objects kept in the > * per cpu partial lists of a processor. > @@ -3989,24 +4012,22 @@ static void set_cpu_partial(struct kmem_cache *s) > * filled up again with minimal effort. The slab will never hit the > * per node partial lists and therefore no locking will be required. > * > - * This setting also determines > - * > - * A) The number of objects from per cpu partial slabs dumped to the > - * per node list when we reach the limit. > - * B) The number of objects in cpu partial slabs to extract from the > - * per node list when we run out of per cpu objects. We only fetch > - * 50% to keep some capacity around for frees. > + * For backwards compatibility reasons, this is determined as number > + * of objects, even though we now limit maximum number of pages, see > + * slub_set_cpu_partial() > */ > if (!kmem_cache_has_cpu_partial(s)) > - slub_set_cpu_partial(s, 0); > + nr_objects = 0; > else if (s->size >= PAGE_SIZE) > - slub_set_cpu_partial(s, 2); > + nr_objects = 2; > else if (s->size >= 1024) > - slub_set_cpu_partial(s, 6); > + nr_objects = 6; > else if (s->size >= 256) > - slub_set_cpu_partial(s, 13); > + nr_objects = 13; > else > - slub_set_cpu_partial(s, 30); > + nr_objects = 30; > + > + slub_set_cpu_partial(s, nr_objects); > #endif > } > > @@ -5379,7 +5400,12 @@ SLAB_ATTR(min_partial); > > static ssize_t cpu_partial_show(struct kmem_cache *s, char *buf) > { > - return sysfs_emit(buf, "%u\n", slub_cpu_partial(s)); > + unsigned int nr_partial = 0; > +#ifdef CONFIG_SLUB_CPU_PARTIAL > + nr_partial = s->cpu_partial; > +#endif > + > + return sysfs_emit(buf, "%u\n", nr_partial); > } > > static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf, > @@ -5450,12 +5476,12 @@ static ssize_t slabs_cpu_partial_show(struct kmem_cache *s, char *buf) > > page = slub_percpu_partial(per_cpu_ptr(s->cpu_slab, cpu)); > > - if (page) { > + if (page) > pages += page->pages; > - objects += page->pobjects; > - } > } > > + /* Approximate half-full pages , see slub_set_cpu_partial() */ > + objects = (pages * oo_objects(s->oo)) / 2; > len += sysfs_emit_at(buf, len, "%d(%d)", objects, pages); > > #ifdef CONFIG_SMP > @@ -5463,9 +5489,12 @@ static ssize_t slabs_cpu_partial_show(struct kmem_cache *s, char *buf) > struct page *page; > > page = slub_percpu_partial(per_cpu_ptr(s->cpu_slab, cpu)); > - if (page) > + if (page) { > + pages = READ_ONCE(page->pages); > + objects = (pages * oo_objects(s->oo)) / 2; > len += sysfs_emit_at(buf, len, " C%d=%d(%d)", > - cpu, page->pobjects, page->pages); > + cpu, objects, pages); > + } > } > #endif > len += sysfs_emit_at(buf, len, "\n"); >
