On 3/31/20 12:13 PM, Toke Høiland-Jørgensen wrote:
Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> writes:
So you install your libxdp-based firewalls and are happy. Then you
decide to install this awesome packet analyzer, which doesn't know
about libxdp yet. Suddenly, you get all packets analyzer, but no more
firewall, until users somehow notices that it's gone. Or firewall
periodically checks that it's still runinng. Both not great, IMO, but
might be acceptable for some users, I guess. But imagine all the
confusion for user, especially if he doesn't give a damn about XDP and
other buzzwords, but only needs a reliable firewall :)
Yes, whereas if the firewall is using bpf_link, then the packet analyser
will be locked out and can't do its thing. Either way you end up with a
broken application; it's just moving the breakage. In the case of
Hm... In one case firewall installation reported success and stopped
working afterwards with no notification and user having no clue. In
another, packet analyzer refused to start and reported error to user.
Let's agree to disagree that those are not at all equivalent. To me
silent failure is so much worse, than application failing to start in
the first place.
I sort of agree with both of you that either case is not great. The silent
override we currently have is not great since it can be evicted at any time
but also bpf_link to lock-out other programs at XDP layer is not great either
since there is also huge potential to break existing programs. It's probably
best to discuss on an actual proposal to see the concrete semantics, but my
concerns, assuming I didn't misunderstand or got confused on something along
the way (if so, please let me know), currently are:
- System service XYZ starts to use XDP with bpf_link one day. Somehow this
application gets shipped by default on mainstream distros and starts up
during init, then effectively locking out everyone else that used to use
the hook today "just fine" given they owned / orchestrated the underlying
networking on the host namespace for the nodes they manage (and for that
it worked before). Now such latter app somehow needs to work around this
breakage by undoing the damage that XYZ did in order to be able to operate
again. There was mentioned 'human override'. I presume whatever it will be,
it will also be done by applications when they don't have another choice.
Otherwise we need to go and tell users that XDP is now only _entirely_
reserved for system service XYZ if you run distro ABC, but not for everyone
else anymore; what answer is there to this? From a PoV where one owns the
entire distro and ecosystem, this is fine, but where this is not the case
as in the rest of the world having to rely on mainstream distros, what is
the answer to users (and especially "those that don't give a damn about XDP,
but just want to get stuff to work" that used to work before, even if we
think silent override is broken)? If the answer is to just 'shrug' and tell
'sorry that's the new way it is right now', then apps will try to use whatever
'human override' there is, and we're back to square one. To provide a
concrete example: if Cilium was configured to load some of its programs on
XDP hook, it currently replaces whatever it was there before. The assumption
is, that in the scenario we're in, we can orchestrate the hostns networking
just fine on K8s nodes since there is just one CNI plugin taking care of that
(and that generally also holds true for the other hooks we're using today).
Now, while we could switch to bpf_link as well and implement it in iproute2
for this specific case, what if someone else starting up earlier and locks
our stuff out? How would we work around it?
- Assuming we have XDP with bpf_link in place with the above, now applications
are forced to start using bpf_link in order to not be locked out by others
using bpf_link as otherwise their application would break. So they need to
support the "old" way of attaching programs as we have today for older
kernels and need to support the bpf_link attachment for newer kernels since
they cannot rely on the old / existing API anymore. There is also a world
outside of C/C++ and thus libbpf / lib{xdp,dispatcher} or whatever, so the
whole rest of the ecosystem is forced to implement it as well due to breakage
concerns, understandable, but quite a burden.
- Equally, in case of Toke's implementation for the cmpxchg-like mechanism in
XDP itself, what happens if an application uses this API and assuming the
library would return the error to the application using it if the expected
program is not attached? Then the application would go for a forceful override
with the existing API today or would it voluntarily bail out and refusing to
work if some other non-cooperating application was loaded in the meantime?
What is the cmpxchg-like mechanism then solving realistically? (And again,
please keep all in mind we cannot force the entire world to use one single
library to rule 'em all, there are plenty of other language runtimes out in
the wild that cannot just import C/C++.)
Thoughts?
Thanks,
Daniel
