On Tue, Feb 02, 2021 at 03:01:22PM -0800, Nick Desaulniers wrote: > > >> Thus far we've been able to successfully reverse engineer it on x86, > > >> though it hasn't been easy. > > >> > > >> There were some particulars for arm64 which made doing so impossible. > > >> (I don't remember the details.) > > > > The main issue is that the tables for arm64 have more indirection than x86. > > I wonder if PAC or BTI also make this slightly more complex? PAC at > least has implications for unwinders, IIUC. What is PAC/BTI? > > On x86, the dispatching jump instruction fetches the target address from > > a contiguous array of addresses based on a given offset. So the list of > > potential targets of the jump is neatly organized in a table (and sure, > > before link time these are just relocation, but still processable). > > > > On arm64 (with GCC at least), what is stored in a table is an array of > > candidate offsets from the jump instruction. And because arm64 is > > limited to 32bit instructions, the encoding often requires multiple > > instructions to compute the target address: > > > > ldr<*> x_offset, [x_offsets_table, x_index, ...] // load offset > > adr x_dest_base, <addr> // load target branch for offset 0 > > add x_dest, x_target_base, x_offset, ... // compute final address > > br x_dest // jump > > > > Where this gets trickier is that (with GCC) the offsets stored in the > > table might or might not be signed constants (and this can be seen in > > GCC intermediate representations, but I do not believe this information > > is output in the final object file). And on top of that, GCC might > > decide to use offsets that are seen as unsigned during intermediate > > representation as signed offset by sign extending them in the add > > instruction. > > > > So, to handle this we'd have to track the different operation done with > > the offset, from the load to the final jump, decoding the instructions > > and deducing the potential target instructions from the table of offsets. > > > > But that is error prone as we don't really know how many instructions > > can be between the ones doing the address computation, and I remember > > some messy case of a jump table inside a jump table where tracking the > > instruction touching one or the other offset would need a lot of corner > > case handling. > > > > And this of course is just for GCC, I haven't looked at what it all > > looks like on Clang's end. > > Sure, but this is what production unwinders do, and they don't require > compiler plugins, right? What do you mean by "production unwinders"? Generally unwinders rely on either frame pointers or DWARF, but (without validation) those aren't robust enough for live patching in the kernel, so I'm not sure how this is relevant. > > > I think the details are pertinent to finding a portable solution. The > > > commit message of this commit in particular doesn't document such > > > details, such as why such an approach is necessary or how the data is > > > laid out for objtool to consume it. > > > > > > > Sorry, I will need to make that clearer. The next patch explains it a > > bit [1] > > > > Basically, for simplicity, the plugin creates a new section containing > > Right, this takes a focus on simplicity, at the cost of alienating a toolchain. > > Ard's point about 3193c0836f20 relating to -fgcse is that when > presented with tricky cases to unwind, the simplest approach is taken. > There it was disabling a compiler specific compiler optimization, here > it's either a compiler specific compiler plugin (or disabling another > compiler optimization). The pattern seems to be "Objtool isn't smart > enough" ... "compiler optimization disabled" or "compiler plugin > dependency." You're taking the two absolute worst case scenarios (one of which is just a patch which doesn't look like it's going to get merged anyway) and drawing a false narrative. In this case the simplest approach would have been to just give up and disable jump tables. We try as hard as possible (beyond turning objtool into a full emulator) to avoid doing that kind of thing because objtool isn't supposed to dictate kernel optimizations. Otherwise we would have disabled jump tables (even for non-retpolines) a long time ago, because that's been a serious PITA. You might not like the plugin -- I don't like it either -- but the goal was to avoid penalizing the kernel with "objtool-friendly" optimizations. That said, jump tables are such a pain for objtool (and currently impossible to deal with for arm64) that I'm completely open to just disabling them if they're shown to have negligible benefit for the kernel. -- Josh
