Currently, without rcu attribute info in BTF, the verifier treats
rcu tagged pointer as a normal pointer. This might be a problem
for sleepable program where rcu_read_lock()/unlock() is not available.
For example, for a sleepable fentry program, if rcu protected memory
access is interleaved with a sleepable helper/kfunc, it is possible
the memory access after the sleepable helper/kfunc might be invalid
since the object might have been freed then. Even without
a sleepable helper/kfunc, without rcu_read_lock() protection,
it is possible that the rcu protected object might be release
in the middle of bpf program execution which may cause incorrect
result.
To prevent above cases, enable btf_type_tag("rcu") attributes,
introduce new bpf_rcu_read_lock/unlock() kfuncs and add verifier support.
In the rest of patch set, Patch 1 enabled btf_type_tag for __rcu
attribute. Patche 2 is a refactoring patch. Patch 3 added new
bpf_rcu_read_lock/unlock() kfuncs and verifier support.
Patch 4 added some tests for these two new kfuncs.
Changelogs:
v6 -> v7:
. rebase on top of bpf-next.
. remove the patch which enables sleepable program using
cgrp_local_storage map. This is orthogonal to this patch set
and will be addressed separately.
. mark the rcu pointer dereference result as UNTRUSTED if inside
a bpf_rcu_read_lock() region.
v5 -> v6:
. fix selftest prog miss_unlock which tested nested locking.
. add comments in selftest prog cgrp_succ to explain how to handle
nested memory access after rcu memory load.
v4 -> v5:
. add new test to aarch64 deny list.
v3 -> v4:
. fix selftest failures when built with gcc. gcc doesn't support
btf_type_tag yet and some tests relies on that. skip these
tests if vmlinux BTF does not have btf_type_tag("rcu").
v2 -> v3:
. went back to MEM_RCU approach with invalidate rcu ptr registers
at bpf_rcu_read_unlock() place.
. remove KF_RCU_LOCK/UNLOCK flag and compare btf_id at verification
time instead.
v1 -> v2:
. use kfunc instead of helper for bpf_rcu_read_lock/unlock.
. not use MEM_RCU bpf_type_flag, instead use active_rcu_lock
in reg state to identify rcu ptr's.
. Add more self tests.
. add new test to s390x deny list.
Yonghong Song (4):
compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))
bpf: Abstract out functions to check sleepable helpers
bpf: Add kfunc bpf_rcu_read_lock/unlock()
selftests/bpf: Add tests for bpf_rcu_read_lock()
include/linux/bpf.h | 3 +
include/linux/bpf_lsm.h | 6 +
include/linux/bpf_verifier.h | 2 +
include/linux/compiler_types.h | 3 +-
include/linux/trace_events.h | 8 +
kernel/bpf/bpf_lsm.c | 20 +-
kernel/bpf/btf.c | 3 +
kernel/bpf/helpers.c | 12 +
kernel/bpf/verifier.c | 156 ++++++++-
kernel/trace/bpf_trace.c | 22 +-
tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 +
tools/testing/selftests/bpf/DENYLIST.s390x | 1 +
.../selftests/bpf/prog_tests/rcu_read_lock.c | 153 +++++++++
.../selftests/bpf/progs/rcu_read_lock.c | 325 ++++++++++++++++++
14 files changed, 695 insertions(+), 20 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c
create mode 100644 tools/testing/selftests/bpf/progs/rcu_read_lock.c
--
2.30.2
