This patch series adds an eBPF JIT for MIPS32. The approach taken fixes and
updates existing code to support MIPS64/MIPS32 systems, then refactors
source into a common core and dependent MIPS64 JIT, and finally adds a
MIPS32 eBPF JIT implementation using the common framework. This approach of
developing MIPS64 and MIPS32 JITs in tandem has desirable benefits for
consistency and long-term maintainability, and the iterative refactoring
has helped identify several problems.
Overview
========
The initial code updates and refactoring exposed a number of problems in
the existing MIPS64 JIT, which are fixed in patches #1 to #9. Patch #10
updates common code to support MIPS64/MIPS32 operation. Patch #12 separates
the common core from the MIPS64 JIT code. Patches #13 and #14 add MIPS64
support for BPF_ATOMIC and BPF_JMP32 insns. Patch #15 adds a needed MIPS32
uasm opcode, while patch #16 adds the MIPS32 eBPF JIT.
Updates to the common core notably include support for bpf2bpf calls and
making tailcalls from BPF subprograms (e.g. patch #11). Some lower priority
features such as MIPS R1 ISA support, direct kernel calls (e.g. for TCP
congestion control experiments) and PROBE_MEM support have been omitted.
On MIPS32, 64-bit BPF registers are mapped to 32-bit register pairs, and
all 64-bit operations are built on 32-bit subregister ops. A few
differences from the MIPS64 JIT include:
* BPF TAILCALL: counter stored on stack due to register pressure.
* BPF_JMP | JSET | BPF_K: drop bbit insns only usable on MIPS64 Octeon
Since MIPS32 does not include 64-bit div/mod or atomic opcodes, these BPF
insns are implemented by directly calling the built-in kernel functions:
(with thanks to Luke Nelson for posting similar code online)
* BPF_ATOMIC | BPF_DW | BPF_ADD (+BPF_FETCH)
* BPF_ATOMIC | BPF_DW | BPF_AND (+BPF_FETCH)
* BPF_ATOMIC | BPF_DW | BPF_XOR (+BPF_FETCH)
* BPF_ATOMIC | BPF_DW | BPF_OR (+BPF_FETCH)
* BPF_ATOMIC | BPF_DW | BPF_XCHG
* BPF_ATOMIC | BPF_DW | BPF_CMPXCHG
* BPF_ALU64 | BPF_DIV | BPF_X
* BPF_ALU64 | BPF_DIV | BPF_K
* BPF_ALU64 | BPF_MOD | BPF_X
* BPF_ALU64 | BPF_MOD | BPF_K
Testing
=======
Testing used LTS kernel 5.10.x and stable 5.13.x running on QEMU/OpenWRT.
The test suite included the 'test_bpf' module, and 'test_verifier' and
'test_progs' from kselftests.
Using 'test_progs' from kselftests proved to be difficult in general
since cross-compilation depends on libbpf/bpftool, which does not support
cross-endian builds. A very hacked build was used, primarily for testing
bpf2bpf calls and tailcalls.
The matrix of test configurations executed for this series covered the
expected register sizes, MIPS ISA releases, and JIT settings:
WORDSIZE={64-bit,32-bit} x ISA={R2,R6} x JIT={off,on,hardened}
On MIPS32BE and MIPS32LE there was general parity between the results of
interpreter vs. JIT-backed tests with respect to the numbers of PASSED,
SKIPPED, and FAILED tests. The same was also true of MIPS64 retesting.
For example, the results below on MIPS32 are typical. Note that skipped
test 885 is a "scale" test which results in OOM on the QEMU malta MIPS32
test systems used.
root@OpenWrt:~# sysctl net.core.bpf_jit_enable=1
root@OpenWrt:~# modprobe test_bpf
...
test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]
root@OpenWrt:~# ./test_verifier 0 884
...
Summary: 1231 PASSED, 0 SKIPPED, 20 FAILED
root@OpenWrt:~# ./test_verifier 886 1184
...
Summary: 459 PASSED, 1 SKIPPED, 2 FAILED
root@OpenWrt:~# ./test_progs -n 105,106
...
105 subprogs:OK
106/1 tailcall_1:OK
106/2 tailcall_2:OK
106/3 tailcall_3:OK
106/4 tailcall_4:OK
106/5 tailcall_5:OK
106/6 tailcall_bpf2bpf_1:OK
106/7 tailcall_bpf2bpf_2:OK
106/8 tailcall_bpf2bpf_3:OK
106/9 tailcall_bpf2bpf_4:OK
106 tailcalls:OK
Summary: 2/9 PASSED, 0 SKIPPED, 0 FAILED
All feedback and suggestions are much appreciated!
---
Change History:
rfc v2:
* Implement all BPF_ATOMIC ops. For MIPS32 BPF_DW insns, call built-in
64-bit kernel functions.
* Add MIPS64 support for BPF_JMP32 conditionals.
* Support making tailcalls from bpf2bpf functions.
* Support bpf2bpf calls with an extra JIT pass to patch call addresses.
* Add JIT support for bpf_line_info via bpf_prog_fill_jited_linfo().
* Further code optimizations, cleanup and simplification.
* Update kernel docs.
rfc v1:
* Initial code proposal, focused on consistency and maintainability for
both MIPS32/MIPS64.
* Several MIPS64 bugfixes and factoring out common shareable code.
* Addition of MIPS32 JIT, roughly matching MIPS64 capabilities.
---
Tony Ambardar (16):
MIPS: eBPF: support BPF_TAIL_CALL in JIT static analysis
MIPS: eBPF: mask 32-bit index for tail calls
MIPS: eBPF: fix BPF_ALU|ARSH handling in JIT static analysis
MIPS: eBPF: support BPF_JMP32 in JIT static analysis
MIPS: eBPF: fix system hang with verifier dead-code patching
MIPS: eBPF: fix JIT static analysis hang with bounded loops
MIPS: eBPF: fix MOD64 insn on R6 ISA
MIPS: eBPF: support long jump for BPF_JMP|EXIT
MIPS: eBPF: drop src_reg restriction in BPF_LD|BPF_DW|BPF_IMM
MIPS: eBPF: add core support for 32/64-bit systems
bpf: allow tailcalls in subprograms for MIPS64/MIPS32
MIPS: eBPF: refactor common 32/64-bit functions and headers
MIPS: eBPF64: support BPF_JMP32 conditionals
MIPS: eBPF64: implement all BPF_ATOMIC ops
MIPS: uasm: Enable muhu opcode for MIPS R6
MIPS: eBPF: add MIPS32 JIT
Documentation/admin-guide/sysctl/net.rst | 6 +-
Documentation/networking/filter.rst | 6 +-
arch/mips/Kconfig | 4 +-
arch/mips/include/asm/uasm.h | 1 +
arch/mips/mm/uasm-mips.c | 4 +-
arch/mips/mm/uasm.c | 3 +-
arch/mips/net/Makefile | 8 +-
arch/mips/net/ebpf_jit.c | 1938 ----------------------
arch/mips/net/ebpf_jit.h | 297 ++++
arch/mips/net/ebpf_jit_comp32.c | 1398 ++++++++++++++++
arch/mips/net/ebpf_jit_comp64.c | 1085 ++++++++++++
arch/mips/net/ebpf_jit_core.c | 1193 +++++++++++++
arch/x86/net/bpf_jit_comp.c | 6 +
include/linux/filter.h | 1 +
kernel/bpf/core.c | 6 +
kernel/bpf/verifier.c | 3 +-
16 files changed, 4010 insertions(+), 1949 deletions(-)
delete mode 100644 arch/mips/net/ebpf_jit.c
create mode 100644 arch/mips/net/ebpf_jit.h
create mode 100644 arch/mips/net/ebpf_jit_comp32.c
create mode 100644 arch/mips/net/ebpf_jit_comp64.c
create mode 100644 arch/mips/net/ebpf_jit_core.c
--
2.25.1
