On Wed, Apr 10, 2019 at 11:50:49AM -0400, Joe Lawrence wrote: > Hi folks, > > This is the third installment of the klp-convert tool for generating and > processing livepatch symbols for livepatch module builds. For those > following along at home, archive links to previous versions: > > RFC: > https://lore.kernel.org/lkml/cover.1477578530.git.jpoimboe@xxxxxxxxxx/ > v2: > https://lore.kernel.org/lkml/f52d29f7-7d1b-ad3d-050b-a9fa8878faf2@xxxxxxxxxx/ > > (Note that I don't see v2 archived at lore, but that is a link to the > most recent subthread that lore did catch.) > > > Livepatches may use symbols which are not contained in its own scope, > and, because of that, may end up compiled with relocations that will > only be resolved during module load. Yet, when the referenced symbols are > not exported, solving this relocation requires information on the object > that holds the symbol (either vmlinux or modules) and its position inside > the object, as an object may contain multiple symbols with the same name. > Providing such information must be done accordingly to what is specified > in Documentation/livepatch/module-elf-format.txt. > > Currently, there is no trivial way to embed the required information as > requested in the final livepatch elf object. klp-convert solves this > problem in two different forms: (i) by relying on a symbol map, which is > built during kernel compilation, to automatically infer the relocation > targeted symbol, and, when such inference is not possible (ii) by using > annotations in the elf object to convert the relocation accordingly to > the specification, enabling it to be handled by the livepatch loader. > > Given the above, add support for symbol mapping in the form of > Symbols.list file; add klp-convert tool; integrate klp-convert tool into > kbuild; make livepatch modules discernible during kernel compilation > pipeline; add data-structure and macros to enable users to annotate > livepatch source code; make modpost stage compatible with livepatches; > update livepatch-sample and update documentation. > > The patch was tested under three use-cases: > > use-case 1: There is a relocation in the lp that can be automatically > resolved by klp-convert. For example. see the saved_command_line > variable in lib/livepatch/test_klp_convert2.c. > > use-case 2: There is a relocation in the lp that cannot be automatically > resolved, as the name of the respective symbol appears in multiple > objects. The livepatch contains an annotation to enable a correct > relocation. See the KLP_MODULE_RELOC / KLP_SYMPOS annotation sections > in lib/livepatch/test_klp_convert{1,2}.c. > > use-case 3: There is a relocation in the lp that cannot be automatically > resolved similarly as 2, but no annotation was provided in the > livepatch, triggering an error during compilation. Reproducible by > removing the KLP_MODULE_RELOC / KLP_SYMPOS annotation sections in > lib/livepatch/test_klp_convert{1,2}.c. > > > Branches > -------- > > Since I spent some time tinkering with v2 and accumulated a bunch of > fixes, I collected them up and am posting this new version. For review > purposes, I posted two branches up to github: > > 1 - an expanded branch that with changes separate from the original > https://github.com/joe-lawrence/linux/commits/klp-convert-v3-expanded > > 2 - a squashed branch of (1) that comprises v3: > https://github.com/joe-lawrence/linux/commits/klp-convert-v3 > > Non-trivial commits in the expanded branch have some extra commentary > and details for debugging in the commit message that were dropped when > squashing into their respective parent commits. > > > TODO > ---- > > There are a few outstanding items that I came across while reviewing v2, > but as changes started accumulating, it made sense to defer those to a > later version. I'll reply to this thread individual topics to start > discussion sub-threads for those. > > > Changelogs > ---------- > > The commit changelogs were getting long, so I've moved them here for > posterity and review purposes: > > livepatch: Create and include UAPI headers > v2: > - jmoreira: split up and changelog > v3: > - joe: convert to SPDX license tags > > kbuild: Support for Symbols.list creation > v3: > - jmoreira: adjust for multiobject livepatch > - joe: add klpclean to PHONY > - joe: align KLP prefix > > livepatch: Add klp-convert tool > v2: > - khlebnikov: use HOSTLOADLIBES_ instead of HOSTLDFLAGS: -lelf must be > at the end > - jmoreira: add support to automatic relocation conversion in > klp-convert.c, changelog > v3: > - joe: convert to SPDX license tags > - jmoreira: add rela symbol name to WARNs > - jmoreira: ignore relocations to .TOC and symbols with index 0 > - joe: separate and fix valid_sympos() sympos=0 and sympos=1..n checks > - joe: fix symbol use-after-frees > - joe: fix remaining valgrind leak complaints (non-error paths only) > - joe: checkpatch nits > > livepatch: Add klp-convert annotation helpers > v2: > - jmoreira: split up: move KLP_MODULE_RELOC from previous patch to > here, add KLP_SYMPOS, move macros from include/uapi/livepatch.h to > include/linux/livepatch.h > v3: > - joe: suggested by jpoimboe, KLP_MODULE_RELOC macro should 4-byte > align klp_module_reloc structures > > modpost: Integrate klp-convert > v2: > - khlebnikov: save cmd_ld_ko_o into .module.cmd, if_changed_rule > doesn't do that.f > - khlebnikov: fix bashisms for debian where /bin/sh is a symlink to > /bin/dash > - khlebnikov: rename rule_link_module to rule_ld_ko_o, otherwise > arg-check inside if_changed_rule compares cmd_link_module and > cmd_ld_ko_o. > - khlebnikov: check modinfo -F livepatch only if CONFIG_LIVEPATCH is > true. > - mbenes: remove modinfo call. LIVEPATCH_ in Makefile > - jmoreira: split up: move the .livepatch file-based scheme for > identifying livepatches to a previous patch, as it was required for > correctly building Symbols.list there. > v3: > - joe: adjust rule_ld_ko_o to call echo-cmd > - joe: rebase for v5.1 > - joe: align KLP prefix > > modpost: Add modinfo flag to livepatch modules > v2: > - jmoreira: fix modpost.c (add_livepatch_flag) to update module > structure with livepatch flag and prevent modpost from breaking due to > unresolved symbols > v3: > - joe: adjust modpost.c::get_modinfo() call for v5.0 version > > livepatch: Add sample livepatch module > v3: > - joe: update all in-tree livepatches with LIVEPATCH_* modinfo flag > > documentation: Update on livepatch elf format > v2: > - jmoreira: update module to use KLP_SYMPOS > - jmoreira: Comments on symbol resolution scheme > - jmoreira: Update Makefile > - jmoreira: Remove MODULE_INFO statement > - jmoreira: Changelog > v3: > - joe: rebase for v5.1 > - joe: code shuffle to better match original source file > > livepatch/selftests: add klp-convert > v3: > - joe: clarify sympos=0 and sympos=1..n > > > And now the usual git cover-letter boilerplate... > > Joao Moreira (2): > kbuild: Support for Symbols.list creation > documentation: Update on livepatch elf format > > Joe Lawrence (1): > livepatch/selftests: add klp-convert > > Josh Poimboeuf (5): > livepatch: Create and include UAPI headers > livepatch: Add klp-convert tool > livepatch: Add klp-convert annotation helpers > modpost: Integrate klp-convert > livepatch: Add sample livepatch module > > Miroslav Benes (1): > modpost: Add modinfo flag to livepatch modules > > .gitignore | 1 + > Documentation/livepatch/livepatch.txt | 3 + > Documentation/livepatch/module-elf-format.txt | 50 +- > MAINTAINERS | 2 + > Makefile | 30 +- > include/linux/livepatch.h | 13 + > include/uapi/linux/livepatch.h | 22 + > kernel/livepatch/core.c | 4 +- > lib/livepatch/Makefile | 15 + > lib/livepatch/test_klp_atomic_replace.c | 1 - > lib/livepatch/test_klp_callbacks_demo.c | 1 - > lib/livepatch/test_klp_callbacks_demo2.c | 1 - > lib/livepatch/test_klp_convert1.c | 106 +++ > lib/livepatch/test_klp_convert2.c | 103 +++ > lib/livepatch/test_klp_convert_mod_a.c | 25 + > lib/livepatch/test_klp_convert_mod_b.c | 13 + > lib/livepatch/test_klp_livepatch.c | 1 - > samples/livepatch/Makefile | 7 + > .../livepatch/livepatch-annotated-sample.c | 102 +++ > samples/livepatch/livepatch-sample.c | 1 - > scripts/Kbuild.include | 4 +- > scripts/Makefile | 1 + > scripts/Makefile.build | 7 + > scripts/Makefile.modpost | 24 +- > scripts/livepatch/.gitignore | 1 + > scripts/livepatch/Makefile | 7 + > scripts/livepatch/elf.c | 753 ++++++++++++++++++ > scripts/livepatch/elf.h | 72 ++ > scripts/livepatch/klp-convert.c | 692 ++++++++++++++++ > scripts/livepatch/klp-convert.h | 39 + > scripts/livepatch/list.h | 391 +++++++++ > scripts/mod/modpost.c | 82 +- > scripts/mod/modpost.h | 1 + > .../selftests/livepatch/test-livepatch.sh | 64 ++ > 34 files changed, 2616 insertions(+), 23 deletions(-) > create mode 100644 include/uapi/linux/livepatch.h > create mode 100644 lib/livepatch/test_klp_convert1.c > create mode 100644 lib/livepatch/test_klp_convert2.c > create mode 100644 lib/livepatch/test_klp_convert_mod_a.c > create mode 100644 lib/livepatch/test_klp_convert_mod_b.c > create mode 100644 samples/livepatch/livepatch-annotated-sample.c > create mode 100644 scripts/livepatch/.gitignore > create mode 100644 scripts/livepatch/Makefile > create mode 100644 scripts/livepatch/elf.c > create mode 100644 scripts/livepatch/elf.h > create mode 100644 scripts/livepatch/klp-convert.c > create mode 100644 scripts/livepatch/klp-convert.h > create mode 100644 scripts/livepatch/list.h > > -- > 2.20.1 [ cc += Jason and Evgenii ] Apologies for the long brain dump, but I promised to reply to this thread with some of the larger TODO issues I came across with this patchset. Static key support is probably the most complicated item, so there is a lot of debugging output I can provide. See the tl;dr and Future Work sections for the executive summary. Static key support ================== tl;dr ----- Livepatch symbols are created when building livepatch modules that reference the (non-exported) static keys of a target object. When loading a livepatch that contains klp-converted static key symbols, the module loader crashes. Testing setup ------------- The easiest way to see the source and repro is to grab this branch: https://github.com/joe-lawrence/linux/commits/klp-convert-v4-static-keys and note these last two commits: - livepatch/klp-convert: abort on static key conversion: this will prevent klp-convert from converting any relocations to the __jump_table. Revert this commit to see the crash. If we don't have a fix before merging, I would suggest a temporary abort like this to avoid silently creating dangerous livepatches. - livepatch/selftests: add livepatch static keys: this adds the self-test which will repro the crash. I added a new livepatch selftest to verify klp-convert and static key behavior: it consists of two kernel modules: test_klp_static_keys_mod.ko and livepatch module that patches it, test_klp_static_keys.ko. The base module contains a few different types of static keys: default true / false, exported / not-exported, and one created by the trace event macro infrastructure. The livepatch module references each of these, relying upon klp-convert. This module also provides its own static key for reference. test_klp_static_keys_mod.ko --------------------------- module_key (false) - exported module_key2 (true) <---- TRACE_EVENT(test_klp_static_keys) <-- | | | | | klp-convert resolves test_klp_static_keys.ko - livepatch | | these references ----------------------------------- | | klp_key (true) | | test_klp_static_keys ----------------- | module_key2 ---------------------------- module_key Currently .klp symbols are created as well as a relocation section for those symbols in the their corresponding .text locations. test_klp_static_keys_mod.ko --------------------------- % eu-readelf --symbols lib/livepatch/test_klp_static_keys_mod.ko | \ grep -e '__tracepoint_test_klp_static_keys' -e '\<module_key[2]*\>' 49: 0000000000000010 16 OBJECT LOCAL DEFAULT 44 module_key 65: 0000000000000010 16 OBJECT LOCAL DEFAULT 32 module_key2 96: 0000000000000000 48 OBJECT GLOBAL DEFAULT 40 __tracepoint_test_klp_static_keys test_klp_static_keys.klp.o -------------------------- % eu-readelf --symbols lib/livepatch/test_klp_static_keys.klp.o | \ grep -e '__tracepoint_test_klp_static_keys' -e '\<module_key[2]*\>' 71: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF __tracepoint_test_klp_static_keys 78: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF module_key2 84: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF module_key Lots of __tracepoint_test_klp_static_keys relocations since each module function registers an event when it is called: % eu-readelf --reloc lib/livepatch/test_klp_static_keys.klp.o Relocation section [ 4] '.rela.text' for section [ 3] '.text' at offset 0x2e7d8 contains 88 entries: Offset Type Value Addend Name ... 0x000000000000002c X86_64_32S 000000000000000000 +0 module_key2 ... 0x000000000000007c X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys ... 0x00000000000000d9 X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys ... 0x0000000000000169 X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys ... 0x00000000000001f9 X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys ... 0x000000000000028c X86_64_32S 000000000000000000 +0 module_key ... 0x00000000000002dc X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys ... 0x000000000000038c X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys ... 0x00000000000003eb X86_64_PC32 000000000000000000 +36 __tracepoint_test_klp_static_keys Relocations generated for the __jump_table itself, note that I grouped each jump entry <code, target, key> relocation set: Relocation section [19] '.rela__jump_table' for section [18] '__jump_table' at offset 0x2fb28 contains 30 entries: Offset Type Value Addend Name 000000000000000000 X86_64_PC32 000000000000000000 +12 .text 0x0000000000000004 X86_64_PC32 000000000000000000 +102 .text 0x0000000000000008 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys 0x0000000000000010 X86_64_PC32 000000000000000000 +188 .text 0x0000000000000014 X86_64_PC32 000000000000000000 +195 .text 0x0000000000000018 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys 0x0000000000000020 X86_64_PC32 000000000000000000 +304 .text 0x0000000000000024 X86_64_PC32 000000000000000000 +0 .text.unlikely 0x0000000000000028 X86_64_PC64 000000000000000000 +16 .bss 0x0000000000000030 X86_64_PC32 000000000000000000 +332 .text 0x0000000000000034 X86_64_PC32 000000000000000000 +339 .text 0x0000000000000038 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys 0x0000000000000040 X86_64_PC32 000000000000000000 +448 .text 0x0000000000000044 X86_64_PC32 000000000000000000 +52 .text.unlikely 0x0000000000000048 X86_64_PC64 000000000000000000 +0 module_key 0x0000000000000050 X86_64_PC32 000000000000000000 +476 .text 0x0000000000000054 X86_64_PC32 000000000000000000 +483 .text 0x0000000000000058 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys 0x0000000000000060 X86_64_PC32 000000000000000000 +592 .text 0x0000000000000064 X86_64_PC32 000000000000000000 +104 .text.unlikely 0x0000000000000068 X86_64_PC64 000000000000000000 +1 module_key2 0x0000000000000070 X86_64_PC32 000000000000000000 +620 .text 0x0000000000000074 X86_64_PC32 000000000000000000 +710 .text 0x0000000000000078 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys 0x0000000000000080 X86_64_PC32 000000000000000000 +796 .text 0x0000000000000084 X86_64_PC32 000000000000000000 +886 .text 0x0000000000000088 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys 0x0000000000000090 X86_64_PC32 000000000000000000 +962 .text 0x0000000000000094 X86_64_PC32 000000000000000000 +981 .text 0x0000000000000098 X86_64_PC64 000000000000000000 +8 __tracepoint_test_klp_static_keys test_klp_static_keys.ko ----------------------- Finally klp-convert has transformed a bunch of symbols for us: % eu-readelf --symbols lib/livepatch/test_klp_static_keys.ko | \ grep -e '__tracepoint_test_klp_static_keys' -e '\<module_key[2]*\>' 71: 0000000000000000 0 NOTYPE GLOBAL DEFAULT LOOS+0 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 78: 0000000000000000 0 NOTYPE GLOBAL DEFAULT LOOS+0 .klp.sym.test_klp_static_keys_mod.module_key2,0 84: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF module_key % eu-readelf --reloc lib/livepatch/test_klp_static_keys.ko Relocation section [19] '.rela__jump_table' for section [18] '__jump_table' at offset 0x2480 contains 30 entries: Offset Type Value Addend Name 000000000000000000 X86_64_PC32 000000000000000000 +12 .text 0x0000000000000004 X86_64_PC32 000000000000000000 +102 .text 0x0000000000000008 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000010 X86_64_PC32 000000000000000000 +188 .text 0x0000000000000014 X86_64_PC32 000000000000000000 +195 .text 0x0000000000000018 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000020 X86_64_PC32 000000000000000000 +304 .text 0x0000000000000024 X86_64_PC32 000000000000000000 +0 .text.unlikely 0x0000000000000028 X86_64_PC64 000000000000000000 +16 .bss 0x0000000000000030 X86_64_PC32 000000000000000000 +332 .text 0x0000000000000034 X86_64_PC32 000000000000000000 +339 .text 0x0000000000000038 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000040 X86_64_PC32 000000000000000000 +448 .text 0x0000000000000044 X86_64_PC32 000000000000000000 +52 .text.unlikely 0x0000000000000048 X86_64_PC64 000000000000000000 +0 module_key 0x0000000000000050 X86_64_PC32 000000000000000000 +476 .text 0x0000000000000054 X86_64_PC32 000000000000000000 +483 .text 0x0000000000000058 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000060 X86_64_PC32 000000000000000000 +592 .text 0x0000000000000064 X86_64_PC32 000000000000000000 +104 .text.unlikely 0x0000000000000068 X86_64_PC64 000000000000000000 +1 .klp.sym.test_klp_static_keys_mod.module_key2,0 0x0000000000000070 X86_64_PC32 000000000000000000 +620 .text 0x0000000000000074 X86_64_PC32 000000000000000000 +710 .text 0x0000000000000078 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000080 X86_64_PC32 000000000000000000 +796 .text 0x0000000000000084 X86_64_PC32 000000000000000000 +886 .text 0x0000000000000088 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000090 X86_64_PC32 000000000000000000 +962 .text 0x0000000000000094 X86_64_PC32 000000000000000000 +981 .text 0x0000000000000098 X86_64_PC64 000000000000000000 +8 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 And here's our final set of klp-converted relocations which include the unexported trace point symbol and module_key2 symbols: Relocation section [44] '.klp.rela.test_klp_static_keys_mod..text' for section [ 3] '.text' at offset 0x55c18 contains 12 entries: Offset Type Value Addend Name ... 0x00000000000003eb X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x000000000000038c X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x00000000000002dc X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x00000000000001f9 X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x0000000000000169 X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x00000000000000d9 X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x000000000000007c X86_64_PC32 000000000000000000 +36 .klp.sym.test_klp_static_keys_mod.__tracepoint_test_klp_static_keys,0 0x000000000000002c X86_64_32S 000000000000000000 +0 .klp.sym.test_klp_static_keys_mod.module_key2,0 ... Current behavior ---------------- Not good. The livepatch successfully builds but crashes on load: % insmod lib/livepatch/test_klp_static_keys_mod.ko % insmod lib/livepatch/test_klp_static_keys.ko BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 #PF error: [normal kernel read fault] PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 3 PID: 9367 Comm: insmod Tainted: G E K 5.1.0-rc4+ #4 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180724_192412-buildhw-07.phx2.fedoraproject.org-1.fc29 04/01/2014 RIP: 0010:jump_label_apply_nops+0x3b/0x60 Code: 02 00 00 48 c1 e5 04 48 01 dd 48 39 eb 74 3a 72 0b eb 36 48 83 c3 10 48 39 dd 76 2d 48 8b 43 08 48 89 c2 83 e0 01 48 83 e2 fc <48> 8b 54 13 10 83 e2 01 38 c2 75 dd 48 89 df 31 f6 48 83 c3 10 e8 RSP: 0018:ffffa8874068fcf8 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffffffffc07fd000 RCX: 000000000000000d RDX: 000000003f803000 RSI: ffffffffa5077be0 RDI: ffffffffc07fe540 RBP: ffffffffc07fd0a0 R08: ffffa88740f43878 R09: ffffa88740eed000 R10: 0000000000055a4b R11: ffffa88740f43878 R12: ffffa88740f430b8 R13: 0000000000000000 R14: ffffa88740f42df8 R15: 0000000000042b01 FS: 00007f4f1dafb740(0000) GS:ffff9a81fbb80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000010 CR3: 00000000b5d8a000 CR4: 00000000000006e0 Call Trace: module_finalize+0x184/0x1c0 load_module+0x1400/0x1910 ? kernel_read_file+0x18d/0x1c0 ? __do_sys_finit_module+0xa8/0x110 __do_sys_finit_module+0xa8/0x110 do_syscall_64+0x55/0x1a0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f4f1cae82bd Future work ----------- At the very least, I think this call-chain ordering is wrong for livepatch static key symbols: load_module apply_relocations post_relocation module_finalize jump_label_apply_nops << ... prepare_coming_module blocking_notifier_call_chain(&module_notify_list, MODULE_STATE_COMING, mod); jump_label_module_notify case MODULE_STATE_COMING jump_label_add_module << do_init_module do_one_initcall(mod->init) __init patch_init [kpatch-patch] klp_register_patch klp_init_patch klp_for_each_object(patch, obj) klp_init_object klp_init_object_loaded klp_write_object_relocations << blocking_notifier_call_chain(&module_notify_list, MODULE_STATE_LIVE, mod); jump_label_module_notify case MODULE_STATE_LIVE jump_label_invalidate_module_init where klp_write_object_relocations() is called way *after* jump_label_apply_nops() and jump_label_add_module(). Detection --------- I have been tinkering with some prototype code to defer jump_label_apply_nops() and jump_label_add_module(), but it has been slow going. I think the jist of it is that we're going to need to call these dynamically when individual klp_objects are patched, not when the livepatch module itself loads. If anyone with static key expertise wants to jump in here, let me know. In the meantime, I cooked up a potential followup commit to detect conversion of static key symbols and klp-convert failure. It basically runs through the output .ko's ELF symbols and verifies that none of the converted ones can be found as a .rela__jump_table relocated symbol. It accurately catches the problematic references in test_klp_static_keys.ko thus far. This was based on a similar issue reported as a bug against kpatch-build, in which Josh wrote code to detect this scenario: https://github.com/dynup/kpatch/issues/946 https://github.com/jpoimboe/kpatch/commit/2cd2d27607566aee9590c367e615207ce1ce24c6 I can post ("livepatch/klp-convert: abort on static key conversion") here as a follow commit if it looks reasonable and folks wish to review it... or we can try and tackle static keys before merging klp-convert. -- Joe
