On Tue, Jan 28, 2025 at 3:08 PM Martin Kelly <martin.kelly@xxxxxxxxxxxxxxx> wrote: > > On Fri, 2025-01-24 at 10:31 -0800, Andrii Nakryiko wrote: > > > On Wed, Jan 22, 2025 at 1:53 PM Slava Imameev > > > <slava.imameev@xxxxxxxxxxxxxxx> wrote: > > > > > > > > > > BPF programs designated as dynamically loaded can be loaded and > > > > > attached independently after the initial bpf_object loading and > > > > > attaching. > > > > > > > > > > These programs can also be reloaded and reattached multiple > > > > > times, > > > > > enabling more flexible management of a resident BPF program > > > > > set. > > > > > > > > > > A key motivation for this feature is to reduce load times for > > > > > utilities that include hundreds of BPF programs. When the > > > > > selection > > > > > of a resident BPF program set cannot be determined at the time > > > > > of > > > > > bpf_object loading and attaching, all BPF programs would > > > > > otherwise > > > > > need to be marked as autoload, leading to unnecessary overhead. > > > > > This patch addresses that inefficiency. > > > > > > Can you elaborate on why it's impossible to determine which BPF > > > programs should be loaded before BPF object load step? > > > > > The main use case for this patch is large applications that need to > dynamically load/unload BPF programs. Our specific use case is a > continuously-running security application with a dynamically- > reconfigurable feature set. As part of that reconfiguration, BPF > programs may get loaded/unloaded on-the-fly. Restarting the entire > application during reconfiguration is undesirable, as critical state > data can be lost and loading hundreds of BPF programs is time- > consuming. Thanks for the details (and sorry for late reply, been traveling lately)! I don't want to complicate bpf_object internals with this third autoload state, as this creates tons of non-obvious gotchas that have to constantly be checked whenever any new feature is added to bpf_object (or even during refactorings). It does seem like you have a pretty complicated use case, and so maybe some of the alternatives would be just fine for you. I see two ways forward for you. Either you can break apart your BPF object of ~100 BPF programs into more independent BPF objects (seeing that programs can be independently loaded/unloaded depending on configuration, seems like you do have a bunch of logic independence, right?). I assume shared BPF maps are the biggest reason to keep all those programs together in one BPF object. To share BPF maps between multiple BPF objects libbpf provides two complementary interfaces: - bpf_map__reuse_fd() for manual control - BPF map pinning (could be declarative or manual) This way you can ensure that all BPF objects would use the same BPF map, where necessary. Alternatively, we can look at this problem as needing libbpf to only prepare BPF program code (doing all the relocations and stuff like that), but then application actually taking care of loading/unloading BPF program with bpf_prog_load() outside of bpf_object abstraction. I've had an almost ready patches splitting bpf_object__load() into two steps: bpf_object__prepare() and bpf_object__load() after that. "prepare" step would create BPF maps, load BTF information, perform necessary relocations and arrive at final state of BPF program code (which you can get with bpf_program__insns() API), but stopping just short of actually doing bpf_prog_load() step. This seems like it would solve your problem as well. You'd use libbpf to do all the low-level ELF processing and relocation, but then take over managing BPF program lifetime. Loading/unloading as you see fit, including in parallel. Is this something that would work for you? > > The above points apply more generically to *any* application that > requires dynamic loading/unloading. Reconfiguration can be done via a > restart, but that has drawbacks: > (a) Losing any non-persistent application state on restart. In our > case, this creates a lapse in security that could be exploited by > adversaries. > (b) In applications with many programs, load+attach can take a long > time. We measured load+attach of ~100 BPF programs taking ~10 seconds > when done with current libbpf serially. Dynamically loading only the > programs needed avoids wasting memory and CPU cycles. > (c) The application itself might take a long time to restart, separate > from the BPF load/attach time. By loading dynamically, the BPF programs > can take effect much sooner and avoid wasted restart cycles. > > This patch set also permits loading BPF programs in parallel if the > application wishes. We tested parallel loading with 200+ BPF programs > and found the load time dropped from 18 seconds to 5 seconds when done > in parallel on a 6.8 kernel. bpf_object is intentionally single-threaded, so I don't think we'll be supporting parallel BPF program loading in the paradigm of bpf_object (but see the bpf_object__prepare() proposal). Even from API standpoint this is problematic with logging and log buffers basically assuming single-threaded execution of BPF program loading. All that could be changed or worked around, but your use case is not really a typical case, so I'm a bit hesitant at this point. > > In the future, this approach could also allow maps to not be > autoloaded, further saving on memory if no program needs the underlying > map. > > In summary, we believe dynamic loading of BPF programs is an important > capability that will improve the performance of CrowdStrike's security > applications as well as being useful to other applications that want to > avoid restarts. > >
