From: Alexei Starovoitov <ast@xxxxxxxxxx> Introduce any context BPF specific memory allocator. Tracing BPF programs can attach to kprobe and fentry. Hence they run in unknown context where calling plain kmalloc() might not be safe. Front-end kmalloc() with per-cpu cache of free elements. Refill this cache asynchronously from irq_work. v1->v2: - Moved unsafe direct call_rcu() from hash map into safe place inside bpf_mem_alloc. Patches 7 and 9. - Optimized atomic_inc/dec in hash map with percpu_counter. Patch 6. - Tuned watermarks per allocation size. Patch 8 - Adopted this approach to per-cpu allocation. Patch 10. - Fully converted hash map to bpf_mem_alloc. Patch 11. - Removed tracing prog restriction on map types. Combination of all patches and final patch 12. v1 thread: https://lore.kernel.org/bpf/20220623003230.37497-1-alexei.starovoitov@xxxxxxxxx/ LWN article: https://lwn.net/Articles/899274/ There is a lot more work ahead, but this set is useful base. Future work: - allow sleepable bpf progs use dynamically allocated maps - expose bpf_mem_alloc as uapi FD to be used in dynptr_alloc, kptr_alloc - add sysctl to force bpf_mem_alloc in hash map when safe even if pre-alloc requested to reduce memory consumption - convert lru map to bpf_mem_alloc Alexei Starovoitov (12): bpf: Introduce any context BPF specific memory allocator. bpf: Convert hash map to bpf_mem_alloc. selftests/bpf: Improve test coverage of test_maps samples/bpf: Reduce syscall overhead in map_perf_test. bpf: Relax the requirement to use preallocated hash maps in tracing progs. bpf: Optimize element count in non-preallocated hash map. bpf: Optimize call_rcu in non-preallocated hash map. bpf: Adjust low/high watermarks in bpf_mem_cache bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU. bpf: Add percpu allocation support to bpf_mem_alloc. bpf: Convert percpu hash map to per-cpu bpf_mem_alloc. bpf: Remove tracing program restriction on map types include/linux/bpf_mem_alloc.h | 26 + kernel/bpf/Makefile | 2 +- kernel/bpf/hashtab.c | 120 ++-- kernel/bpf/memalloc.c | 638 ++++++++++++++++++++++ kernel/bpf/syscall.c | 5 +- kernel/bpf/verifier.c | 29 - samples/bpf/map_perf_test_kern.c | 44 +- samples/bpf/map_perf_test_user.c | 2 +- tools/testing/selftests/bpf/progs/timer.c | 11 - tools/testing/selftests/bpf/test_maps.c | 38 +- 10 files changed, 811 insertions(+), 104 deletions(-) create mode 100644 include/linux/bpf_mem_alloc.h create mode 100644 kernel/bpf/memalloc.c -- 2.30.2
