At Meta we have a profiling daemon which periodically collects
information on many hosts. This collection usually involves grabbing
stacks (user and kernel) using perf_event BPF progs and later symbolicating
them. For user stacks we try to use BPF_F_USER_BUILD_ID and rely on
remote symbolication, but BPF_F_USER_BUILD_ID doesn't always succeed. In
those cases we must fall back to digging around in /proc/PID/maps to map
virtual address to (binary, offset). The /proc/PID/maps digging does not
occur synchronously with stack collection, so the process might already
be gone, in which case it won't have /proc/PID/maps and we will fail to
symbolicate.
This 'exited process problem' doesn't occur very often as
most of the prod services we care to profile are long-lived daemons, but
there are enough usecases to warrant a workaround: a BPF program which
can be optionally loaded at data collection time and essentially walks
/proc/PID/maps. Currently this is done by walking the vma list:
struct vm_area_struct* mmap = BPF_CORE_READ(mm, mmap);
mmap_next = BPF_CORE_READ(rmap, vm_next); /* in a loop */
Since commit 763ecb035029 ("mm: remove the vma linked list") there's no
longer a vma linked list to walk. Walking the vma maple tree is not as
simple as hopping struct vm_area_struct->vm_next. Luckily,
commit f39af05949a4 ("mm: add VMA iterator"), another commit in that series,
added struct vma_iterator and for_each_vma macro for easy vma iteration. If
similar functionality was exposed to BPF programs, it would be perfect for our
usecase.
This series adds such functionality, specifically a BPF equivalent of
for_each_vma using the open-coded iterator style.
Notes:
* This approach was chosen after discussion on a previous series [0] which
attempted to solve the same problem by adding a BPF_F_VMA_NEXT flag to
bpf_find_vma.
* Unlike the task_vma bpf_iter, the open-coded iterator kfuncs here do not
drop the vma read lock between iterations. See Alexei's response in [0].
* The [vsyscall] page isn't really part of task->mm's vmas, but
/proc/PID/maps returns information about it anyways. The vma iter added
here does not do the same. See comment on selftest in patch 3.
* The struct vma_iterator wrapped by struct bpf_iter_task_vma itself wraps
struct ma_state. Because we need the entire struct, not a ptr, changes to
either struct vma_iterator or struct ma_state will necessitate changing the
opaque struct bpf_iter_task_vma to account for the new size. This feels a
bit brittle. We could instead use bpf_mem_alloc to allocate a struct
vma_iterator in bpf_iter_task_vma_new and have struct bpf_iter_task_vma
point to that, but that's not quite equivalent as BPF progs will usually
use the stack for this struct via bpf_for_each. Went with the simpler route
for now.
Patch summary:
* Patch 1 is a tiny fix I ran into while implementing the vma iter in this
series. It can be applied independently.
* Patch 2 is the meat of the implementation
* Patch 3 adds tests for the new functionality
* Existing iter tests exercise failure cases (e.g. prog that doesn't call
_destroy()). I didn't replicate them in this series, but am happy to add
them in v2 if folks feel that it would be worthwhile.
[0]: https://lore.kernel.org/bpf/20230801145414.418145-1-davemarchevsky@xxxxxx/
Dave Marchevsky (3):
bpf: Explicitly emit BTF for struct bpf_iter_num, not btf_iter_num
bpf: Introduce task_vma open-coded iterator kfuncs
selftests/bpf: Add tests for open-coded task_vma iter
include/uapi/linux/bpf.h | 5 ++
kernel/bpf/bpf_iter.c | 2 +-
kernel/bpf/helpers.c | 3 +
kernel/bpf/task_iter.c | 54 ++++++++++++++
tools/include/uapi/linux/bpf.h | 5 ++
tools/lib/bpf/bpf_helpers.h | 8 +++
.../selftests/bpf/prog_tests/bpf_iter.c | 26 +++----
.../testing/selftests/bpf/prog_tests/iters.c | 71 +++++++++++++++++++
...f_iter_task_vma.c => bpf_iter_task_vmas.c} | 0
.../selftests/bpf/progs/iters_task_vma.c | 56 +++++++++++++++
10 files changed, 216 insertions(+), 14 deletions(-)
rename tools/testing/selftests/bpf/progs/{bpf_iter_task_vma.c => bpf_iter_task_vmas.c} (100%)
create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma.c
--
2.34.1
