Starting with 5.14 kernels, I can reliably reproduce a crash [1] on
ppc64le when loading livepatches containing late klp-relocations [2].
These are relocations, specific to livepatching, that are resolved not
when a livepatch module is loaded, but only when a livepatch-target
module is loaded.
There was previously related work by Josh and Peter [3] to simplify a
lot of x86 and s390x code (at the time, the only two arches to
HAVE_LIVEPATCH and STRICT_MODULE_RWX) as part of disallowing writable
executable mappings.
Now that Power has STRICT_MODULE_RWX, I think we will need to consider
this architecture as well.
The crash was originally spotted by the external kpatch-build tool [4]
when building its integration tests on rhel-9-beta. It can also be
reproduced by the endless-WIP klp-convert patchset [5], which brings
klp-relocation creation from kpatch-build to the upstream build.
I further verified:
- turning STRICT_MODULE_RWX off resulted in no crash
- alternatively, reverting the following commits resulted in no crash:
d556e1be3332 ("livepatch: Remove module_disable_ro() usage")
0d9fbf78fefb ("module: Remove module_disable_ro()")
I haven't started looking at a fix yet, but in the case of the x86 code
update, its apply_relocate_add() implementation was modified to use a
common text_poke() function to allowed us to drop
module_{en,dis}ble_ro() games by the livepatching code.
I can take a closer look this week, but thought I'd send out a report
in case this may be a known todo for STRICT_MODULE_RWX on Power.
-- Joe
[1] crashing kernel log
[ 84.837986] ===== TEST: klp-convert symbols =====
[ 84.858937] % modprobe test_klp_convert_mod
[ 84.879040] % modprobe test_klp_convert1
[ 84.908056] BUG: Unable to handle kernel data access on write at 0xc0080000018402f0
[ 84.908067] Faulting instruction address: 0xc000000000056b58
[ 84.908072] Oops: Kernel access of bad area, sig: 11 [#1]
[ 84.908077] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[ 84.908082] Modules linked in: test_klp_convert1(K+) test_klp_convert_mod bonding tls rfkill pseries_rng drm fuse drm_panel_orientation_quirks xfs libcrc32c sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto dm_mirror dm_region_hash dm_log dm_mod [last unloaded: test_klp_atomic_replace]
[ 84.908114] CPU: 1 PID: 4205 Comm: modprobe Kdump: loaded Tainted: G K 5.14.0+ #2
[ 84.908121] NIP: c000000000056b58 LR: c000000000056b1c CTR: 0000000000000009
[ 84.908127] REGS: c00000005dce3480 TRAP: 0300 Tainted: G K (5.14.0+)
[ 84.908132] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24224484 XER: 00000000
[ 84.908147] CFAR: c000000000056a68 DAR: c0080000018402f0 DSISR: 0a000000 IRQMASK: 0
GPR00: c000000000056b1c c00000005dce3720 c000000002a2af00 0000000000000000
GPR04: c0080000018402f0 396b00003d620000 e98b0020f8410018 00000000ffffffff
GPR08: 4e8004207d8903a6 0000000080000000 c0080000018382f0 000000000000000d
GPR12: 0000000000004000 c000000007fcf480 c00000004d7e2000 c0080000018706d8
GPR16: c008000001850228 c00000004d7e2c00 00000000ffffffff c0000000010d6248
GPR20: c00000000298c1c8 c008000001860380 c0080000018706f0 aaaaaaaaaaaaaaab
GPR24: c00000004d7e2b40 c008000001870000 c00800000184005c 000000000000008c
GPR28: c008000001860380 c008000000770008 c00000004d7e2000 c0080000018402f0
[ 84.908209] NIP [c000000000056b58] create_stub+0x78/0x240
[ 84.908217] LR [c000000000056b1c] create_stub+0x3c/0x240
[ 84.908223] Call Trace:
[ 84.908225] [c00000005dce3720] [c00000004d7e2b40] 0xc00000004d7e2b40 (unreliable)
[ 84.908232] [c00000005dce37a0] [c000000000056e0c] stub_for_addr+0xec/0x120
[ 84.908240] [c00000005dce37d0] [c000000000057f14] apply_relocate_add+0x814/0x9a0
[ 84.908247] [c00000005dce38d0] [c00000000021ca38] klp_apply_section_relocs+0x208/0x2d0
[ 84.908255] [c00000005dce39c0] [c00000000021cb90] klp_init_object_loaded+0x90/0x1d0
[ 84.908262] [c00000005dce3a50] [c00000000021d2dc] klp_enable_patch+0x32c/0x540
[ 84.908269] [c00000005dce3b10] [c008000001840030] test_klp_convert_init+0x28/0x48 [test_klp_convert1]
[ 84.908277] [c00000005dce3b30] [c000000000012230] do_one_initcall+0x60/0x2c0
[ 84.908284] [c00000005dce3c00] [c00000000026012c] do_init_module+0x7c/0x3b0
[ 84.908290] [c00000005dce3c90] [c000000000262b74] __do_sys_finit_module+0xd4/0x160
[ 84.908296] [c00000005dce3db0] [c000000000030664] system_call_exception+0x144/0x280
[ 84.908303] [c00000005dce3e10] [c00000000000bff0] system_call_vectored_common+0xf0/0x280
[ 84.908310] --- interrupt: 3000 at 0x7fffa06d6b9c
[ 84.908315] NIP: 00007fffa06d6b9c LR: 0000000000000000 CTR: 0000000000000000
[ 84.908320] REGS: c00000005dce3e80 TRAP: 3000 Tainted: G K (5.14.0+)
[ 84.908325] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28224244 XER: 00000000
[ 84.908340] IRQMASK: 0
GPR00: 0000000000000161 00007fffc4f74ad0 00007fffa07d7100 0000000000000005
GPR04: 000000012a926ca0 0000000000000000 0000000000000005 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffa0f9c380 0000000000000020 0000000000000000
GPR16: 00000100010a1de0 0000000000000000 000000012a927d50 00000100010a02f8
GPR20: 0000000000000001 0000000000000908 00000100010a2020 00000100010a19b0
GPR24: 0000000000000000 0000000000000000 00000100010a2040 00000100010a03f0
GPR28: 00000100010a1e00 000000012a926ca0 0000000000040000 00000100010a19b0
[ 84.908399] NIP [00007fffa06d6b9c] 0x7fffa06d6b9c
[ 84.908403] LR [0000000000000000] 0x0
[ 84.908406] --- interrupt: 3000
[ 84.908410] Instruction dump:
[ 84.908413] 3d02ffb2 395f8000 3d208000 3ce0ffff 38c68d70 39088d84 79290020 60e7ffff
[ 84.908423] e8a60014 e8c80008 e9080010 78e70020 <f8bf0000> f8df0008 f91f0010 811c0224
[ 84.908435] ---[ end trace 961b4b817da4a53b ]---
[2] https://www.kernel.org/doc/html/latest/livepatch/module-elf-format.html
[3] https://lore.kernel.org/lkml/cover.1588173720.git.jpoimboe@xxxxxxxxxx/
[4] https://github.com/dynup/kpatch/issues/1228
[5] https://github.com/joe-lawrence/linux/tree/klp-convert-v5-expanded-v5.14-rebase1
