On Tue, Mar 31, 2020 at 6:49 AM Daniel Borkmann <daniel@xxxxxxxxxxxxx> wrote:
>
> On 3/31/20 12:13 PM, Toke Høiland-Jørgensen wrote:
> > Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> writes:
> >
> >>>> So you install your libxdp-based firewalls and are happy. Then you
> >>>> decide to install this awesome packet analyzer, which doesn't know
> >>>> about libxdp yet. Suddenly, you get all packets analyzer, but no more
> >>>> firewall, until users somehow notices that it's gone. Or firewall
> >>>> periodically checks that it's still runinng. Both not great, IMO, but
> >>>> might be acceptable for some users, I guess. But imagine all the
> >>>> confusion for user, especially if he doesn't give a damn about XDP and
> >>>> other buzzwords, but only needs a reliable firewall :)
> >>>
> >>> Yes, whereas if the firewall is using bpf_link, then the packet analyser
> >>> will be locked out and can't do its thing. Either way you end up with a
> >>> broken application; it's just moving the breakage. In the case of
> >>
> >> Hm... In one case firewall installation reported success and stopped
> >> working afterwards with no notification and user having no clue. In
> >> another, packet analyzer refused to start and reported error to user.
> >> Let's agree to disagree that those are not at all equivalent. To me
> >> silent failure is so much worse, than application failing to start in
> >> the first place.
>
> I sort of agree with both of you that either case is not great. The silent
> override we currently have is not great since it can be evicted at any time
> but also bpf_link to lock-out other programs at XDP layer is not great either
> since there is also huge potential to break existing programs. It's probably
I disagree with the premise that in current setup two XDP applications
can work at all. Best case, one will fail if specified the option to
not attach if something is already attached. Worst case, both will
happily assume (and report to user) that they are working, but only
the one attached last will actually work. Or maybe it's not even the
worst scenario, if both of them use netlink notification and keep
re-attaching themselves and detaching "opponent" (a fun bot fight to
watch...)
So what I hope we are discussing here is the world where some
applications are moving into using libxdp or some other co-operative
approach/daemon. In that case, utmost importance (otherwise its
unreliable and half-working solution) is to prevent XDP applications
not aware about this cooperation approach to break cooperating ones.
And that can be done only if libxdp/daemon can guarantee that *if it
successfully* attaches root XDP program, it won't be replaced by
oblivious XDP application that is not aware of it. So yes, in this
case non-cooperating application won't work (as it might not with
current API), but at least it will report that it can't attach, as
opposed to break *all* other nicely behaving and cooperating XDP
appplications **silently**. There are probably few more frustrating
things than silent corruption/breakage, IMO as both a user and
programmer.
> best to discuss on an actual proposal to see the concrete semantics, but my
> concerns, assuming I didn't misunderstand or got confused on something along
> the way (if so, please let me know), currently are:
>
> - System service XYZ starts to use XDP with bpf_link one day. Somehow this
> application gets shipped by default on mainstream distros and starts up
> during init, then effectively locking out everyone else that used to use
> the hook today "just fine" given they owned / orchestrated the underlying
> networking on the host namespace for the nodes they manage (and for that
If XYZ didn't use bpf_link and just used existing API, it would break
everything as well, because see above, they can't co-exist. The
difference is in amount of undeterminism (who starts first and who's
second) and awareness (whether app even knows that it's broken).
Neither are in favor of existing API.
> it worked before). Now such latter app somehow needs to work around this
> breakage by undoing the damage that XYZ did in order to be able to operate
> again. There was mentioned 'human override'. I presume whatever it will be,
> it will also be done by applications when they don't have another choice.
No, it's opposite. That's why it's **human override**. It's not
intended to be used by application to "unblock" itself.
> Otherwise we need to go and tell users that XDP is now only _entirely_
> reserved for system service XYZ if you run distro ABC, but not for everyone
> else anymore; what answer is there to this? From a PoV where one owns the
That's exactly what Toke's libxdp is intending to be. An XDP
coordination solution/library that other applications that want to
co-exist on the same network interface **have** to use. It intends to
allow all XDP applications to co-exist. That would be the answer - go
use that library.
> entire distro and ecosystem, this is fine, but where this is not the case
> as in the rest of the world having to rely on mainstream distros, what is
> the answer to users (and especially "those that don't give a damn about XDP,
> but just want to get stuff to work" that used to work before, even if we
> think silent override is broken)? If the answer is to just 'shrug' and tell
> 'sorry that's the new way it is right now', then apps will try to use whatever
> 'human override' there is, and we're back to square one. To provide a
> concrete example: if Cilium was configured to load some of its programs on
> XDP hook, it currently replaces whatever it was there before. The assumption
> is, that in the scenario we're in, we can orchestrate the hostns networking
> just fine on K8s nodes since there is just one CNI plugin taking care of that
> (and that generally also holds true for the other hooks we're using today).
> Now, while we could switch to bpf_link as well and implement it in iproute2
> for this specific case, what if someone else starting up earlier and locks
> our stuff out? How would we work around it?
For one, you can use some tool/script (like the one I posted
yesterday: [0]) to find "offending" application that's not expected to
be using XDP and kill it (and/or investigate why on earth it got
installed/started in your infrastructure without you being aware). I
think this solution is better than nuke ("human override") option,
because it gives you clues on what's misbehaving and needs fixing in
the first place.
[0] https://gist.github.com/anakryiko/562dff8e39c619a5ee247bb55aa057c7
>
> - Assuming we have XDP with bpf_link in place with the above, now applications
> are forced to start using bpf_link in order to not be locked out by others
> using bpf_link as otherwise their application would break. So they need to
> support the "old" way of attaching programs as we have today for older
> kernels and need to support the bpf_link attachment for newer kernels since
> they cannot rely on the old / existing API anymore. There is also a world
> outside of C/C++ and thus libbpf / lib{xdp,dispatcher} or whatever, so the
> whole rest of the ecosystem is forced to implement it as well due to breakage
> concerns, understandable, but quite a burden.
Multi-XDP (on the same netdev, of course) doesn't exist and is not
possible today with existing API and semantics. The world outside of
C/C++ will either need to use compatible mechanisms (linking and using
libxdp with whatever means their language/runtime provides or at least
re-implementing the same set of protocols and behaviors).
>
> - Equally, in case of Toke's implementation for the cmpxchg-like mechanism in
> XDP itself, what happens if an application uses this API and assuming the
> library would return the error to the application using it if the expected
> program is not attached? Then the application would go for a forceful override
> with the existing API today or would it voluntarily bail out and refusing to
> work if some other non-cooperating application was loaded in the meantime?
> What is the cmpxchg-like mechanism then solving realistically? (And again,
> please keep all in mind we cannot force the entire world to use one single
> library to rule 'em all, there are plenty of other language runtimes out in
> the wild that cannot just import C/C++.)
It's not about using one specific library, but it is about following
the same protocol. I think that's what Toke, Alexei, Andrey and others
are assuming - that yes, if people want to write reliable XDP
applications co-existing with each other, they will have to use the
same library (easier, if possible) or at least follow the same
protocol.
>
> Thoughts?
>
> Thanks,
> Daniel
