From: Hou Tao <houtao1@xxxxxxxxxx>
Hi,
The motivation of inlining bpf_kptr_xchg() comes from the performance
profiling of bpf memory allocator benchmark [1]. The benchmark uses
bpf_kptr_xchg() to stash the allocated objects and to pop the stashed
objects for free. After inling bpf_kptr_xchg(), the performance for
object free on 8-CPUs VM increases about 2%~10%. However the performance
gain comes with costs: both the kasan and kcsan checks on the pointer
will be unavailable. Initially the inline is implemented in do_jit() for
x86-64 directly, but I think it will more portable to implement the
inline in verifier.
Patch #1 supports inlining bpf_kptr_xchg() helper and enables it on
x86-4. Patch #2 factors out a helper for newly-added test in patch #3.
Patch #3 tests whether the inlining of bpf_kptr_xchg() is expected.
Please see individual patches for more details. And comments are always
welcome.
Change Log:
v2:
* rebased on bpf-next tree
* drop patch #1 in v1 due to discussion in [2]
* patch #1: add the motivation in the commit message, merge patch #1
and #3 into the new patch in v2. (Daniel)
* patch #2/#3: newly-added patch to test the inlining of
bpf_kptr_xchg() (Eduard)
v1: https://lore.kernel.org/bpf/95b8c2cd-44d5-5fe1-60b5-7e8218779566@xxxxxxxxxxxxxxx/
[1]: https://lore.kernel.org/bpf/20231221141501.3588586-1-houtao@xxxxxxxxxxxxxxx/
[2]: https://lore.kernel.org/bpf/fd94efb9-4a56-c982-dc2e-c66be5202cb7@xxxxxxxxxxxxxxx/
Hou Tao (3):
bpf: Support inlining bpf_kptr_xchg() helper
selftests/bpf: Factor out get_xlated_program() helper
selftests/bpf: Test the inlining of bpf_kptr_xchg()
arch/x86/net/bpf_jit_comp.c | 5 ++
include/linux/filter.h | 1 +
kernel/bpf/core.c | 10 ++++
kernel/bpf/helpers.c | 1 +
kernel/bpf/verifier.c | 17 +++++++
.../selftests/bpf/prog_tests/ctx_rewrite.c | 44 ----------------
.../bpf/prog_tests/kptr_xchg_inline.c | 51 +++++++++++++++++++
.../selftests/bpf/progs/kptr_xchg_inline.c | 28 ++++++++++
tools/testing/selftests/bpf/test_verifier.c | 47 +----------------
tools/testing/selftests/bpf/testing_helpers.c | 42 +++++++++++++++
tools/testing/selftests/bpf/testing_helpers.h | 6 +++
11 files changed, 163 insertions(+), 89 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c
create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_inline.c
--
2.29.2
