Hi, the notes I've taken during the sessions today are below. They are verbatim as taken during the session, please apologize any odd formulations or typos you encounter. Please don't forget to upload your slides to the LPC webpage; you should be able to attach them to your proposal. Alternatively, just send them to me via e-mail and I'll add them there. Thanks! ========== 1) gcc optimizations gcc makes different optimizations during compilation. Depending how the patches are being prepared, this might or might not be a problem. As of gcc 6.2, the especially problematic ones seem to be: - inlining; solution: patch callers - partial inlining - IPA-SRA: removal of unused parameters, passing by reference -> passing by value; basically violates ABI. Fortunately, gcc changes the name of function (-> easily detectable) - IPA-CP: even several clones of a function are possible if a set is limited again, function name is changed again by gcc - IPA-PURE-CONST: calls to such functions might be completely eliminated - IPA-ICF: can merge different functions to unified one - dead code elimination: after patch the code wouldn't be dead any more solution: patch the callers as well - IPA-RA: gcc optimizes the saving and restoring registers if it considers it safe; we should probably disabled 2) Module creation tooling Current aproaches are kpatch-build (out-of-tree) and kbuild (in-tree). kpatch-build is based on comparing binary objects before and after applying the patch. It has quite a few drabacks though (a lot of internal knowledge of gcc and kernel is required to handle it properly -- altinstructions, exception tables, switch statement jump tables, static local variables, etc). Plus it's a maintainance nightmare. kbuild-based aproach is based on providing a new whole reimplementation of the function in a kernel module and register it with klp API. One of the main drawbacks is dealing with non-exported symbols. The current workardound is to perform kallsyms lookup on the symbol, but that has its own issues (error prone, doesn't use the relocation sections). Proposal (sent as RFC last week) is a new tool on top of the kbuild-based aproach (klp-convert) that will be run to post-process the linked .ko and generate proper relovation entries. Improvements that would need to be done on top would be - support for altinstructions (we may need to change klp API for that) - warning on unresolved symbols - automatically detect and convert all relocations (all unresolved relocations in the module) - automatically detect and warn about gcc optimizations Chris O'Donell brought up a pending gcc modification (in the form of plugin) that will contain range-based information about the optimizations that were performed while generating the code; which would be very helpful for our use case as well. 3) Patch module dependencies The principal question is whether we would support (and if so, how) the patches for modules that haven't been yet loaded. Are there any benefits of enforcing hard module dependencies for module? Currently, livepatch can patch multiple objects at once including unloaded modules. This is achieved by delaying certain operations to module-load time. The issues this aproach has: - code duplication (especially with module loader) - ELF information needs to be preserved The question to be answered is whether there is any better aproach to this problem. What if we split up modules per object? What if we enforce hard module dependencies? Pros: - no need to split up rela sections per object (we still need it for rela.sym) - the same for arch sections - no need for duplicate code in arch_klp_init_object_loaded() - no need to carry ELF information over - removal of module "notifier" calls The open question is how we'll be tracking module dependencies. Is it possible via modprobe? Steven Rostedt suggested that we instead patch on-disk modules instead. This has been discussed in the past, and not done at the end on the basis of: - maintainance nightmare (both from RPM POV and "what is in-memory" POV) - reverting modules becomes difficult - what goes into initrd? Steven: is reverting really important? Yes, for testing fixes, for being able to disable broken fix (and applying fix on top is not a maintainable alternative, also the user is not able to generate the on-top fix himself) 4) Brainstorming session for applying patches that change data structure The ultimate goal is to be able to apply patches that - change semantics of global variable - change semantics of structure member - add/remove struct member The major problems are: how to find the data, how perform the changes, how to resize the structure. Jiri stated that he's afraid that general solution might not very well be possible, because we're highly depending on the semantics. Josh said that they had success with using the shadow structure aproach; at the same time they updated the code that worked with the struct. Miroslav stated that given his past experience shows that there might not be significant number of CVEs that woudl require it. Sasha suggested to use padding in structs which is usually there as a workaround in specific cases. 5) Patching scheduler functions Patching schedule() is a tricky operation, as it returns with a different stack. New schedule() generated by compiler might be completely different. Hence if a task enters new schedule() and returns with a stack that has been set up by the old schedule(), explosion happens. Steven suggested to make the %rip saving conditional on livepatching being turned on, and doesn't see an issue with that. Vlastimil noted that upon starting of the live patching (and enabling the static branch), all the running tasks would need to have their thread_ip updated in task struct. The conclusion has quickly been reached that we would pursue this upstream. 6) Livepatching support for different archs Petr summarized pre-requisities for basic support of the live patching, which are: - ftrace handler at the very beginning of the function prologue - FTRACE_WITH_REGS support powerpc64le needs special boilerplate code for handling TOC (v2 ABI) ARM64 is in similar situation, but the ABI is saner so the solution might be easier. 7) Prioritizing next steps upstream The stack reliability work is mostly done upstream, so we should now revive the frame pointer testing and verification. Also the objtool is being updated not to require the stack pointer. The conclusion has been reached that it's proper time now to revive the hybrid consistency model patchset, as the stack unwinder is now in a proper shape (modulo the frame pointer requirement). 8) Live patching -- review Balbir described the specifics of PPC64 ABI and its specifics with respect to live patching (especially the intrinsics of TOC and link register) and described the technical details on how FTRACE_WITH_REGS has been implemented on ppc64le so that it's compatible with ABIv2. Balbir then brought up the need for testing the live patching infrastructure (as a lot of it is in assembly, it's hard to formally validated) and specifically need for comprehensive suite. Then a discussion evolved around security implications and distribution channels for the patch modules. ========== -- Jiri Kosina SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe live-patching" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
