"Brian G. Merrell" <brian.g.merrell@xxxxxxxxx> writes: > Hi all, > > tl;dr: What does the future look like for Go programs orchestrating > multiple, chained eBPF network functions? Is it worth considering doing > the orchestration from C++ (or Rust) instead of Go? I'm hoping the > Cilium and/or Katran folks (and any other interested people) can weigh > in! Thank you for bringing this up! Adding a few people (and bpf@vger) to Cc to widen the discussion a bit. > -- > > I just joined a team working with the orchestration of multiple XDP (and > TC) programs. By "orchestration" I mean that various eBPF programs can > be written by multiple teams for different systems and we have logic to > decide how to run, update, and chain them together. This seems to be a > fast-moving topic right now and I'm trying to get my bearings as well as > prepare for the future. Feel free to just point me to relevant docs or > code > > This is essentially what we do today: We have a top level Golang > orchestration program that has access to a database that contains all > the business logic (e.g., variable values for the bpf programs depending > on datacenter), the latest build of the C BPF userspace, and kernel > programs. Basically, like this: > > +--> [userspace prog 1] --> [kern prog 1] > | > [Go orchestrator] +--> [userspace prog 2] --> [kern prog 2] > | > +--> [userspace prog 3] --> [kern prog 3] > > The Go program simply executes (fork+exec) the userspace programs with > the appropriate command-line arguments for their environment. The > userspace program loads the kernel programs, which need to do a > bpf_tail_call to the next program, which is exposed with some bpf map > mojo[1]. > > I think it's not too dissimilar from what the Cilium and Katran folks > have been doing. I think our current approach is actually pretty cool > considering that the project started a couple of years ago, but I'm > trying to plot a course for the future. > > I have a couple of concerns about the current design: > > 1. The kernel programs need to make the bpf_tail_call. I'd prefer our > internal teams can write network functions without needing to be > aware of other network functions. > 2. The Go orchestrator simply doing a fork+exec feels naughty. I'm > assuming there's potentially important error state information that > we might be missing out on by not working with an library API. > > Regarding #1, Toke Høiland-Jørgensen was kind enough to point me to his > recent work for the Linux 5.10 kernel and xdp-loader (backed by xdplib) > that I think addresses the underlying concern. However, I'm not so sure > how to handle my concern #2. > > I think ideally, our new flow would look something like this: > > +--> [kern prog 1] > | > [Go orchestrator] --> [xdplib] +--> [kern prog 2] > | > +--> [kern prog 3] > > Assuming that make sense, I have a few questions: > * is there any work being done for a Go interface for xdplib? > * interface for xdplib? Any ideas on the level of effort to do that? > > Alternatively, we could probably just execute the xdp-loader binary from > Go, but that that goes back to my concern #2 above. As I see it there are basically four paths to widen the language support for libxdp/multiprog: 1. Regular language bindings for the C libxdp. I gather this is somewhat cumbersome in Go, though, as evidenced by the existence of a native-go BPF library. 2. Reimplement the libxdp functionality in each language. Libxdp really implements a "protocol" for how to cooperatively build a multiprog dispatcher from component programs, including atomic replace etc. This could be re-implemented by other libraries / in other languages as well, and if people want to go this route I'm happy to write up a formal specification if that's helpful. I'm not aware of any efforts in this direction thus far, though. 3. Make xdp-loader explicitly support the fork/exec use case you're describing above. Nothing says this has to lose any information compared to the library, we just have to design for it (JSON output and the ability to pass a BPF object file as an fd on exit would be the main things missing here, I think). I certainly wouldn't object to including this in xdp-loader. 4. Implement an "xdpd" daemon that exposes an IPC interface (say, a socket) and does all the stuff libxdp currently does in application context. Each language then just has to talk to the daemon which is less complex than the full libxdp re-implementation. I've been resisting the daemon approach because I don't like introducing another dependency for running XDP. But my main concern is that we end up with something that is compatible across environments and implementations, so applications that use XDP don't have to deal with multiple incompatible ways of loading their programs. So any feedback anyone has as to which approach(es) would/would not work for you would be welcome (speaking to everyone here)! -Toke
