Hi all,
tl;dr: What does the future look like for Go programs orchestrating
multiple, chained eBPF network functions? Is it worth considering doing
the orchestration from C++ (or Rust) instead of Go? I'm hoping the
Cilium and/or Katran folks (and any other interested people) can weigh
in!
--
I just joined a team working with the orchestration of multiple XDP (and
TC) programs. By "orchestration" I mean that various eBPF programs can
be written by multiple teams for different systems and we have logic to
decide how to run, update, and chain them together. This seems to be a
fast-moving topic right now and I'm trying to get my bearings as well as
prepare for the future. Feel free to just point me to relevant docs or
code
This is essentially what we do today: We have a top level Golang
orchestration program that has access to a database that contains all
the business logic (e.g., variable values for the bpf programs depending
on datacenter), the latest build of the C BPF userspace, and kernel
programs. Basically, like this:
+--> [userspace prog 1] --> [kern prog 1]
|
[Go orchestrator] +--> [userspace prog 2] --> [kern prog 2]
|
+--> [userspace prog 3] --> [kern prog 3]
The Go program simply executes (fork+exec) the userspace programs with
the appropriate command-line arguments for their environment. The
userspace program loads the kernel programs, which need to do a
bpf_tail_call to the next program, which is exposed with some bpf map
mojo[1].
I think it's not too dissimilar from what the Cilium and Katran folks
have been doing. I think our current approach is actually pretty cool
considering that the project started a couple of years ago, but I'm
trying to plot a course for the future.
I have a couple of concerns about the current design:
1. The kernel programs need to make the bpf_tail_call. I'd prefer our
internal teams can write network functions without needing to be
aware of other network functions.
2. The Go orchestrator simply doing a fork+exec feels naughty. I'm
assuming there's potentially important error state information that
we might be missing out on by not working with an library API.
Regarding #1, Toke Høiland-Jørgensen was kind enough to point me to his
recent work for the Linux 5.10 kernel and xdp-loader (backed by xdplib)
that I think addresses the underlying concern. However, I'm not so sure
how to handle my concern #2.
I think ideally, our new flow would look something like this:
+--> [kern prog 1]
|
[Go orchestrator] --> [xdplib] +--> [kern prog 2]
|
+--> [kern prog 3]
Assuming that make sense, I have a few questions:
* is there any work being done for a Go interface for xdplib?
* interface for xdplib? Any ideas on the level of effort to do that?
Alternatively, we could probably just execute the xdp-loader binary from
Go, but that that goes back to my concern #2 above.
Thanks in advance for any feedback; please let me know if I'm even on
the right track with my thoughts here.
Thanks,
Brian
[1] https://www.youtube.com/watch?v=NpDoK6kmGe0&feature=youtu.be&t=2379
|