On Fri 29-05-20 09:06:10, Feng Tang wrote: > When checking a performance change for will-it-scale scalability mmap test > [1], we found very high lock contention for spinlock of percpu counter > 'vm_committed_as': > > 94.14% 0.35% [kernel.kallsyms] [k] _raw_spin_lock_irqsave > 48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap; > 45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap; > > Actually this heavy lock contention is not always necessary. The > 'vm_committed_as' needs to be very precise when the strict > OVERCOMMIT_NEVER policy is set, which requires a rather small batch number > for the percpu counter. > > So keep 'batch' number unchanged for strict OVERCOMMIT_NEVER policy, and > lift it to 64X for OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS policies. Also > add a sysctl handler to adjust it when the policy is reconfigured. > > Benchmark with the same testcase in [1] shows 53% improvement on a 8C/16T > desktop, and 2097%(20X) on a 4S/72C/144T server. We tested with test > platforms in 0day (server, desktop and laptop), and 80%+ platforms shows > improvements with that test. And whether it shows improvements depends on > if the test mmap size is bigger than the batch number computed. > > And if the lift is 16X, 1/3 of the platforms will show improvements, > though it should help the mmap/unmap usage generally, as Michal Hocko > mentioned: > > : I believe that there are non-synthetic worklaods which would benefit from > : a larger batch. E.g. large in memory databases which do large mmaps > : during startups from multiple threads. > > [1] https://lore.kernel.org/lkml/20200305062138.GI5972@shao2-debian/ > > Link: http://lkml.kernel.org/r/1589611660-89854-4-git-send-email-feng.tang@xxxxxxxxx > Signed-off-by: Feng Tang <feng.tang@xxxxxxxxx> > Cc: Michal Hocko <mhocko@xxxxxxxx> > Cc: Matthew Wilcox (Oracle) <willy@xxxxxxxxxxxxx> > Cc: Johannes Weiner <hannes@xxxxxxxxxxx> > Cc: Mel Gorman <mgorman@xxxxxxx> > Cc: Kees Cook <keescook@xxxxxxxxxxxx> > Cc: Andi Kleen <andi.kleen@xxxxxxxxx> > Cc: Tim Chen <tim.c.chen@xxxxxxxxx> > Cc: Dave Hansen <dave.hansen@xxxxxxxxx> > Cc: Huang Ying <ying.huang@xxxxxxxxx> > Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> Acked-by: Michal Hocko <mhocko@xxxxxxxx> > --- > include/linux/mm.h | 2 ++ > include/linux/mman.h | 4 ++++ > kernel/sysctl.c | 2 +- > mm/mm_init.c | 18 ++++++++++++++---- > mm/util.c | 12 ++++++++++++ > 5 files changed, 33 insertions(+), 5 deletions(-) > > diff --git a/include/linux/mm.h b/include/linux/mm.h > index 573947c..c2efea6 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -206,6 +206,8 @@ int overcommit_ratio_handler(struct ctl_table *, int, void *, size_t *, > loff_t *); > int overcommit_kbytes_handler(struct ctl_table *, int, void *, size_t *, > loff_t *); > +int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *, > + loff_t *); > > #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n)) > > diff --git a/include/linux/mman.h b/include/linux/mman.h > index 4b08e9c..91c93c1 100644 > --- a/include/linux/mman.h > +++ b/include/linux/mman.h > @@ -57,8 +57,12 @@ extern struct percpu_counter vm_committed_as; > > #ifdef CONFIG_SMP > extern s32 vm_committed_as_batch; > +extern void mm_compute_batch(void); > #else > #define vm_committed_as_batch 0 > +static inline void mm_compute_batch(void) > +{ > +} > #endif > > unsigned long vm_memory_committed(void); > diff --git a/kernel/sysctl.c b/kernel/sysctl.c > index db1ce7a..9456c86 100644 > --- a/kernel/sysctl.c > +++ b/kernel/sysctl.c > @@ -2650,7 +2650,7 @@ static struct ctl_table vm_table[] = { > .data = &sysctl_overcommit_memory, > .maxlen = sizeof(sysctl_overcommit_memory), > .mode = 0644, > - .proc_handler = proc_dointvec_minmax, > + .proc_handler = overcommit_policy_handler, > .extra1 = SYSCTL_ZERO, > .extra2 = &two, > }, > diff --git a/mm/mm_init.c b/mm/mm_init.c > index 435e5f7..c5a6fb1 100644 > --- a/mm/mm_init.c > +++ b/mm/mm_init.c > @@ -13,6 +13,7 @@ > #include <linux/memory.h> > #include <linux/notifier.h> > #include <linux/sched.h> > +#include <linux/mman.h> > #include "internal.h" > > #ifdef CONFIG_DEBUG_MEMORY_INIT > @@ -144,14 +145,23 @@ EXPORT_SYMBOL_GPL(mm_kobj); > #ifdef CONFIG_SMP > s32 vm_committed_as_batch = 32; > > -static void __meminit mm_compute_batch(void) > +void mm_compute_batch(void) > { > u64 memsized_batch; > s32 nr = num_present_cpus(); > s32 batch = max_t(s32, nr*2, 32); > - > - /* batch size set to 0.4% of (total memory/#cpus), or max int32 */ > - memsized_batch = min_t(u64, (totalram_pages()/nr)/256, 0x7fffffff); > + unsigned long ram_pages = totalram_pages(); > + > + /* > + * For policy of OVERCOMMIT_NEVER, set batch size to 0.4% > + * of (total memory/#cpus), and lift it to 25% for other > + * policies to easy the possible lock contention for percpu_counter > + * vm_committed_as, while the max limit is INT_MAX > + */ > + if (sysctl_overcommit_memory == OVERCOMMIT_NEVER) > + memsized_batch = min_t(u64, ram_pages/nr/256, INT_MAX); > + else > + memsized_batch = min_t(u64, ram_pages/nr/4, INT_MAX); > > vm_committed_as_batch = max_t(s32, memsized_batch, batch); > } > diff --git a/mm/util.c b/mm/util.c > index fe63271..580d268 100644 > --- a/mm/util.c > +++ b/mm/util.c > @@ -746,6 +746,18 @@ int overcommit_ratio_handler(struct ctl_table *table, int write, void *buffer, > return ret; > } > > +int overcommit_policy_handler(struct ctl_table *table, int write, void *buffer, > + size_t *lenp, loff_t *ppos) > +{ > + int ret; > + > + ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); > + if (ret == 0 && write) > + mm_compute_batch(); > + > + return ret; > +} > + > int overcommit_kbytes_handler(struct ctl_table *table, int write, void *buffer, > size_t *lenp, loff_t *ppos) > { > -- > 2.7.4 > -- Michal Hocko SUSE Labs
