Hi Everyone,
This patch implements inlining of calls to bpf_loop helper function
when bpf_loop's callback is statically known. E.g. the rewrite does
the following transformation during BPF program processing:
bpf_loop(10, foo, NULL, 0);
->
for (int i = 0; i < 10; ++i)
foo(i, NULL);
The transformation leads to measurable latency change for simple
loops. Measurements using `benchs/run_bench_bpf_loop.sh` inside QEMU /
KVM on i7-4710HQ CPU shows a drop in latency from 14 ns/op to 2 ns/op.
The change is split in three parts:
* Update to test_verifier.c to specify expected and unexpected
instruction sequences. This allows to check BPF program rewrites
applied by do_mix_fixups function.
* Update to test_verifier.c to specify BTF function infos and types
per test case. This is necessary for tests that load sub-program
addresses to a variable because of the checks applied by
check_ld_imm function.
* The update to verifier.c that tracks state of the parameters for
each bpf_loop call in a program and decides whether it could be
replaced by a loop.
Additional details are available in the commit message for each patch.
Hope you find this useful.
Best regards,
Eduard Zingerman
Eduard Zingerman (3):
selftests/bpf: specify expected instructions in test_verifier tests
selftests/bpf: allow BTF specs and func infos in test_verifier tests
bpf: Inline calls to bpf_loop when callback is known
include/linux/bpf_verifier.h | 15 +
kernel/bpf/bpf_iter.c | 9 +-
kernel/bpf/verifier.c | 160 +++++++++-
.../selftests/bpf/prog_tests/bpf_loop.c | 21 ++
tools/testing/selftests/bpf/prog_tests/btf.c | 1 -
tools/testing/selftests/bpf/progs/bpf_loop.c | 38 +++
tools/testing/selftests/bpf/test_btf.h | 2 +
tools/testing/selftests/bpf/test_verifier.c | 290 +++++++++++++++++-
.../selftests/bpf/verifier/bpf_loop_inline.c | 49 +++
9 files changed, 558 insertions(+), 27 deletions(-)
create mode 100644 tools/testing/selftests/bpf/verifier/bpf_loop_inline.c
--
2.25.1
