Hi Everyone,
This patch implements inlining of calls to bpf_loop helper function
when bpf_loop's callback is statically known. E.g. the rewrite does
the following transformation during BPF program processing:
bpf_loop(10, foo, NULL, 0);
->
for (int i = 0; i < 10; ++i)
foo(i, NULL);
The transformation leads to measurable latency change for simple
loops. Measurements using `benchs/run_bench_bpf_loop.sh` inside QEMU /
KVM on i7-4710HQ CPU show a drop in latency from 14 ns/op to 2 ns/op.
The change is split in three parts:
* Update to test_verifier.c to specify expected and unexpected
instruction sequences. This allows to check BPF program rewrites
applied by do_mix_fixups function.
* Update to test_verifier.c to specify BTF function infos and types
per test case. This is necessary for tests that load sub-program
addresses to a variable because of the checks applied by
check_ld_imm function.
* The update to verifier.c that tracks state of the parameters for
each bpf_loop call in a program and decides whether it could be
replaced by a loop.
Additional details are available in commit messages for each patch.
Hope you will find this useful.
Note: the newly added `bpf_loop/check_stack` test case in `test_progs`
currently fails in no_alu32 mode with error "BPF program is too
large". I'm still investigating this issue. Nevertheless, I'm still
interested in overall feedback for the patch.
Best regards,
Eduard Zingerman
Changes since v1:
- allow to use SKIP_INSNS in instruction pattern specification in
test_verifier tests;
- fix for a bug in spill offset assignement for loop vars when
bpf_loop is located in a non-main function.
Eduard Zingerman (3):
selftests/bpf: specify expected instructions in test_verifier tests
selftests/bpf: allow BTF specs and func infos in test_verifier tests
bpf: Inline calls to bpf_loop when callback is known
include/linux/bpf_verifier.h | 16 +
kernel/bpf/bpf_iter.c | 9 +-
kernel/bpf/verifier.c | 184 +++++++++-
.../selftests/bpf/prog_tests/bpf_loop.c | 62 ++++
tools/testing/selftests/bpf/prog_tests/btf.c | 1 -
tools/testing/selftests/bpf/progs/bpf_loop.c | 122 +++++++
tools/testing/selftests/bpf/test_btf.h | 2 +
tools/testing/selftests/bpf/test_verifier.c | 338 +++++++++++++++++-
.../selftests/bpf/verifier/bpf_loop_inline.c | 244 +++++++++++++
9 files changed, 951 insertions(+), 27 deletions(-)
create mode 100644 tools/testing/selftests/bpf/verifier/bpf_loop_inline.c
--
2.25.1
