From: Alexei Starovoitov <ast@xxxxxxxxxx>
Introduce any context BPF specific memory allocator.
Tracing BPF programs can attach to kprobe and fentry. Hence they
run in unknown context where calling plain kmalloc() might not be safe.
Front-end kmalloc() with per-cpu per-bucket cache of free elements.
Refill this cache asynchronously from irq_work.
There is a lot more work ahead, but this set is useful base.
Future work:
- get rid of call_rcu in hash map
- get rid of atomic_inc/dec in hash map
- tune watermarks per allocation size
- adopt this approach alloc_percpu_gfp
- expose bpf_mem_alloc as uapi FD to be used in dynptr_alloc, kptr_alloc
- add sysctl to force bpf_mem_alloc in hash map when safe even if pre-alloc
requested to reduce memory consumption
- convert lru map to bpf_mem_alloc
Alexei Starovoitov (5):
bpf: Introduce any context BPF specific memory allocator.
bpf: Convert hash map to bpf_mem_alloc.
selftests/bpf: Improve test coverage of test_maps
samples/bpf: Reduce syscall overhead in map_perf_test.
bpf: Relax the requirement to use preallocated hash maps in tracing
progs.
include/linux/bpf_mem_alloc.h | 26 ++
kernel/bpf/Makefile | 2 +-
kernel/bpf/hashtab.c | 16 +-
kernel/bpf/memalloc.c | 512 ++++++++++++++++++++++++
kernel/bpf/verifier.c | 31 +-
samples/bpf/map_perf_test_kern.c | 22 +-
tools/testing/selftests/bpf/test_maps.c | 38 +-
7 files changed, 610 insertions(+), 37 deletions(-)
create mode 100644 include/linux/bpf_mem_alloc.h
create mode 100644 kernel/bpf/memalloc.c
--
2.30.2
