Allow error injection during module initialization for testing.
This adds ALLOW_ERROR_INJECTION() annotation for complete_formation(),
do_init_module() and module_enable_rodata_ro_after_init() so we can test
the error path in these functions.
In addition, add moderr support, an eBPF based tool to inject module
errors in any of the above functions for a specific module name.
The following example is added to the moderr commit. Adding it also here
for convenience:
Example: Inject error -22 to module_enable_rodata_ro_after_init for
brd module:
sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()
Kernel messages:
[ 89.463690] brd: module loaded
[ 89.463855] brd: module_enable_rodata_ro_after_init() returned -22,
ro_after_init data might still be writable
The tool is based on libbpf and is located under tools/bpf.
Here some specific questions for discussion.
* I've noticed that some users of ALLOW_ERROR_INJECTION() in the
kernel add the option to fail behind a Kconfig (Fault-injection
framework, FAULT_INJECTION) in lib/Kconfig.debug, such as FAILSLAB or
FAIL_PAGE_ALLOC. Shall we add similar condition for error injection
in modules?
* What other functions may people be interested in adding to the list of
suported for error injection?
* I found a more generic error injection tool (inject.py) [1] in
IOVisor/BCC (Ccing IOVisor mailing list as well). The tool seems to
be maintained and supports error injection based on a specified call
chain and optional predicates. However, it relies on the old and
deprecated Python eBPF/BCC infrastructure. Since there are other users
of ALLOW_ERROR_INJECTION(), I'm curious: is there a different generic
tool to generate error injection in the kernel already converted to
libbpf? Do users of ALLOW_ERROR_INJECTION() still depend on inject.py
tool?
[1] https://github.com/iovisor/bcc/blob/master/tools/inject.py
Additionally, would it make sense to develop an in-tree tool with
libbpf support for this purpose? This approach might require rewriting
inject.py in libbpf and/or extending moderr to include inject.py
features (probably replace and rework some of the current functionality
in moderr).
moderr uses an enum to detect which kprobe function to override the
return value. I think adding bpf_get_stack()/bpf_get_stackid() support
would avoid the need for hardcoding the kprobe detection mechanism.
This approach could also be used for navigation of the call stack as in
inject.py. Are there any alternative methods to achieve this that avoid
relying on enumerating the fuctions to probe?
Signed-off-by: Daniel Gomez <da.gomez@xxxxxxxxxxx>
---
Daniel Gomez (2):
module: allow for module error injection
moderr: add module error injection tool
kernel/module/main.c | 3 +
kernel/module/strict_rwx.c | 2 +
tools/bpf/Makefile | 13 ++-
tools/bpf/moderr/.gitignore | 2 +
tools/bpf/moderr/Makefile | 95 +++++++++++++++++
tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
tools/bpf/moderr/moderr.c | 236 ++++++++++++++++++++++++++++++++++++++++++
tools/bpf/moderr/moderr.h | 40 +++++++
8 files changed, 515 insertions(+), 3 deletions(-)
---
base-commit: 232f121837ad8b1c21cc80f2c8842a4090c5a2a0
change-id: 20250121-modules-error-injection-176c525791b0
Best regards,
--
Daniel Gomez <da.gomez@xxxxxxxxxxx>
