Hi Everyone,
This is the next iteration of the patch. Please find updated intro
message and change log below.
This patch implements inlining of calls to bpf_loop helper function
when bpf_loop's callback is statically known. E.g. the rewrite does
the following transformation during BPF program processing:
bpf_loop(10, foo, NULL, 0);
->
for (int i = 0; i < 10; ++i)
foo(i, NULL);
The transformation leads to measurable latency change for simple
loops. Measurements using `benchs/run_bench_bpf_loop.sh` inside QEMU /
KVM on i7-4710HQ CPU show a drop in latency from 14 ns/op to 2 ns/op.
The change is split in five parts:
* Update to test_verifier.c to specify expected and unexpected
instruction sequences. This allows to check BPF program rewrites
applied by do_mix_fixups function.
* Update to test_verifier.c to specify BTF function infos and types
per test case. This is necessary for tests that load sub-program
addresses to a variable because of the checks applied by
check_ld_imm function.
* The update to verifier.c that tracks state of the parameters for
each bpf_loop call in a program and decides whether it could be
replaced by a loop.
* A set of test cases for `test_verifier` that use capabilities added
by the first two patches to verify instructions produced by inlining
logic.
* Two test cases for `test_prog` to check that possible corner cases
behave as expected.
Additional details are available in commit messages for each patch.
Changes since v2:
- fix for `stack_check` test case in `test_progs-no_alu32`, all tests
are passing now;
- v2 3/3 patch is split in three parts:
- kernel changes
- test_verifier changes
- test_prog changes
- updated `inline_bpf_loop` in `verifier.c` to calculate each offset
used in instructions to avoid "magic" numbers;
- removed newline handling logic in `fail_log` branch of
`do_single_test` in `test_verifier.c` to simplify the patch set;
- styling fixes suggested in review for v2 of this patch set.
Changes since v1:
- allow to use SKIP_INSNS in instruction pattern specification in
test_verifier tests;
- fix for a bug in spill offset assignement for loop vars when
bpf_loop is located in a non-main function.
Eduard Zingerman (5):
selftests/bpf: specify expected instructions in test_verifier tests
selftests/bpf: allow BTF specs and func infos in test_verifier tests
bpf: Inline calls to bpf_loop when callback is known
selftests/bpf: BPF test_verifier selftests for bpf_loop inlining
selftests/bpf: BPF test_prog selftests for bpf_loop inlining
include/linux/bpf_verifier.h | 16 +
kernel/bpf/bpf_iter.c | 9 +-
kernel/bpf/verifier.c | 199 ++++++++++-
.../selftests/bpf/prog_tests/bpf_loop.c | 61 ++++
tools/testing/selftests/bpf/prog_tests/btf.c | 1 -
tools/testing/selftests/bpf/progs/bpf_loop.c | 114 +++++++
tools/testing/selftests/bpf/test_btf.h | 2 +
tools/testing/selftests/bpf/test_verifier.c | 316 +++++++++++++++++-
.../selftests/bpf/verifier/bpf_loop_inline.c | 244 ++++++++++++++
9 files changed, 935 insertions(+), 27 deletions(-)
create mode 100644 tools/testing/selftests/bpf/verifier/bpf_loop_inline.c
--
2.25.1
