Ard,
Sorry, I was late to the party, attempting to reply to the entire thread
at once. Also, adding the live-patching ML.
I agree with a lot of your concerns. Reverse engineering the control
flow of the compiled binary is kind of ridiculous. I was always
surprised that it works. I still am! But I think it's more robust than
you give it credit for.
Most of the existing code just works, with (annual) tweaks for new
compiler versions. In fact now it works well with both GCC and Clang,
across several versions. Soon it will work with LTO.
It has grown many uses beyond stack validation: ORC, static calls,
retpolines validation, noinstr validation, SMAP validation. It has
found a *lot* of compiler bugs. And there will probably be more use
cases when we get vmlinux validation running fast enough.
But there is indeed a maintenance burden. I often ask myself if it's
worth it. So far the answer has been yes :-) Particularly because it
has influenced many positive changes to the kernel. And it helps now
that even more people are contributing and adding useful features.
But you should definitely think twice before letting it loose on your
arch, especially if you have some other way to ensure reliable stack
metadata, and if you don't have a need for the other objtool features.
Regarding your other proposals:
1) I'm doubtful we can ever rely on the toolchain to ensure reliable
unwind metadata, because:
a) most of the problems are in asm and inline-asm; good luck getting
the toolchain to care about those.
b) the toolchain is fragile; do we want to entrust the integrity of
live patching to the compiler's frame pointer generation (or other
unwind metadata) without having some sort of checking mechanism?
2) The shadow stack idea sounds promising -- how hard would it be to
make a prototype reliable unwinder?
More comments below:
On Thu, Jan 21, 2021 at 12:48:43PM +0100, Ard Biesheuvel wrote:
> On Thu, 21 Jan 2021 at 12:23, Peter Zijlstra <peterz@xxxxxxxxxxxxx> wrote:
> >
> > On Thu, Jan 21, 2021 at 12:08:23PM +0100, Ard Biesheuvel wrote:
> > > On Thu, 21 Jan 2021 at 11:26, Julien Thierry <jthierry@xxxxxxxxxx> wrote:
> >
> > > > I'm not familiar with toolcahin code models, but would this approach be
> > > > able to validate assembly code (either inline or in assembly files?)
> > > >
> > >
> > > No, it would not. But those files are part of the code base, and can
> > > be reviewed and audited.
> >
> > x86 has a long history if failing at exactly that.
>
> That's a fair point. But on the flip side, maintaining objtool does
> not look like it has been a walk in the park either.
I think you missed Peter's point: it's not that it's *hard* for humans
to continuously review/audit all asm and inline-asm; it's just not
feasible to do it 100% correctly, 100% of the time.
Like any other code, objtool requires maintenance, but its analysis is
orders of magnitude more robust than any human.
> What i am especially concerned about is things like 3193c0836f20,
> where we actually have to disable certain compiler optimizations
> because they interfere with objtool's ability to understand the
> resulting object code. Correctness and performance are challenging
> enough as requirements for generated code.
Well, you managed to find the worst case scenario. I think that's the
only time we ever had to do that. Please don't think that's normal, or
even a generally realistic concern. Objtool tries really hard to stay
out of the way.
Long term we really want to prevent that type of thing with the help of
annotations from compiler plugins, similar to what Julien did here.
Yes, it would mean two objtool compiler plugins (GCC and Clang), but it
would ease the objtool maintenance burden and risk in many ways. And
prevent situations like that commit you found. It may sound fragile,
but it will actually make things simpler overall: less reverse
engineering of GCC switch jump tables and __noreturn functions is a good
thing.
> Mind you, I am not saying it is not worth it *for x86*, where there is
> a lot of other stuff going on. But on arm64, we don't care about ORC,
> about -fomit-frame-pointer, about retpolines or about any of the other
> things objtool enables.
>
> On arm64, all it currently seems to provide is a way to capture the
> call stack accurately, and given that it needs a GCC plugin for this
> (which needs to be maintained as well, which is non-trivial, and also
> bars us from using objtool with Clang builds), my current position is
> simply that opening this can of worms at this point is just not worth
> it.
As far as GCC plugins go, it looks pretty basic to me. Also this
doesn't *at all* prevent Clang from being used for live patching. If
anybody actually tries running Julien's patches on a Clang-built kernel,
it might just work. But if not, and the switch tables turn out to be
unparseable like on GCC, we could have a Clang plugin. As I mentioned,
we'll probably end up having one anyway for x86.
--
Josh
