On Fri, 19 Aug 2022 at 23:42, Alexei Starovoitov <alexei.starovoitov@xxxxxxxxx> wrote: > > From: Alexei Starovoitov <ast@xxxxxxxxxx> > > Introduce any context BPF specific memory allocator. > > Tracing BPF programs can attach to kprobe and fentry. Hence they > run in unknown context where calling plain kmalloc() might not be safe. > Front-end kmalloc() with per-cpu cache of free elements. > Refill this cache asynchronously from irq_work. > > Major achievements enabled by bpf_mem_alloc: > - Dynamically allocated hash maps used to be 10 times slower than fully preallocated. > With bpf_mem_alloc and subsequent optimizations the speed of dynamic maps is equal to full prealloc. > - Tracing bpf programs can use dynamically allocated hash maps. > Potentially saving lots of memory. Typical hash map is sparsely populated. > - Sleepable bpf programs can used dynamically allocated hash maps. > >From my side, for the whole series: Acked-by: Kumar Kartikeya Dwivedi <memxor@xxxxxxxxx> > v2->v3: > - Rewrote the free_list algorithm based on discussions with Kumar. Patch 1. > - Allowed sleepable bpf progs use dynamically allocated maps. Patches 13 and 14. > - Added sysctl to force bpf_mem_alloc in hash map even if pre-alloc is > requested to reduce memory consumption. Patch 15. > - Fix: zero-fill percpu allocation > - Single rcu_barrier at the end instead of each cpu during bpf_mem_alloc destruction > > v2 thread: > https://lore.kernel.org/bpf/20220817210419.95560-1-alexei.starovoitov@xxxxxxxxx/ > > v1->v2: > - Moved unsafe direct call_rcu() from hash map into safe place inside bpf_mem_alloc. Patches 7 and 9. > - Optimized atomic_inc/dec in hash map with percpu_counter. Patch 6. > - Tuned watermarks per allocation size. Patch 8 > - Adopted this approach to per-cpu allocation. Patch 10. > - Fully converted hash map to bpf_mem_alloc. Patch 11. > - Removed tracing prog restriction on map types. Combination of all patches and final patch 12. > > v1 thread: > https://lore.kernel.org/bpf/20220623003230.37497-1-alexei.starovoitov@xxxxxxxxx/ > > LWN article: > https://lwn.net/Articles/899274/ > > Future work: > - expose bpf_mem_alloc as uapi FD to be used in dynptr_alloc, kptr_alloc > - convert lru map to bpf_mem_alloc > > Alexei Starovoitov (15): > bpf: Introduce any context BPF specific memory allocator. > bpf: Convert hash map to bpf_mem_alloc. > selftests/bpf: Improve test coverage of test_maps > samples/bpf: Reduce syscall overhead in map_perf_test. > bpf: Relax the requirement to use preallocated hash maps in tracing > progs. > bpf: Optimize element count in non-preallocated hash map. > bpf: Optimize call_rcu in non-preallocated hash map. > bpf: Adjust low/high watermarks in bpf_mem_cache > bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU. > bpf: Add percpu allocation support to bpf_mem_alloc. > bpf: Convert percpu hash map to per-cpu bpf_mem_alloc. > bpf: Remove tracing program restriction on map types > bpf: Prepare bpf_mem_alloc to be used by sleepable bpf programs. > bpf: Remove prealloc-only restriction for sleepable bpf programs. > bpf: Introduce sysctl kernel.bpf_force_dyn_alloc. > > include/linux/bpf_mem_alloc.h | 26 + > include/linux/filter.h | 2 + > kernel/bpf/Makefile | 2 +- > kernel/bpf/core.c | 2 + > kernel/bpf/hashtab.c | 132 +++-- > kernel/bpf/memalloc.c | 601 ++++++++++++++++++++++ > kernel/bpf/syscall.c | 14 +- > kernel/bpf/verifier.c | 52 -- > samples/bpf/map_perf_test_kern.c | 44 +- > samples/bpf/map_perf_test_user.c | 2 +- > tools/testing/selftests/bpf/progs/timer.c | 11 - > tools/testing/selftests/bpf/test_maps.c | 38 +- > 12 files changed, 795 insertions(+), 131 deletions(-) > create mode 100644 include/linux/bpf_mem_alloc.h > create mode 100644 kernel/bpf/memalloc.c > > -- > 2.30.2 >
