On Sun, Mar 14, 2021 at 8:40 AM Gilad Reti <gilad.reti@xxxxxxxxx> wrote:
>
> As libbpf is heading towards a first major release, we wanted to
> discuss libbpf's object pinning strategy.
>
> bpf object pinning has a couple of use cases (feel free to add, there
> are more for sure):
> 1. Sharing specific bpf objects between different processes (for
> example, one process loads a bpf skeleton, another one interacts with
> it using various bpf maps (for example, for changing configurations
> (i.e. dynamic networking rules etc))
> 2. Preventing bpf objects from destruction upon owning process exit
> (i.e. to prevent bpf progs detach upon userspace program crash)
>
> Regarding the first use case, for most cases manually setting the pin
> path (both in the loading process and in other processes) will
> probably be the best. In such cases, no redesign is required here.
>
> For the second one, something like the bpf_object__pin will be more
> appropriate (to allow a complete reuse of the bpf objects). For that
> use case, some sensible requirements we can consider are:
>
> 1. Paths should be unique:
> a. at the bpf_object level (that is, same pinnable objects that
> belong to different bpf_object s should be pinned at different paths).
> b. in the same bpf_object, between different pinnable object types
> (i.e. a map and a prog) should always be pinned at different paths.
> c. different objects, belonging to the same bpf_object and of the
> same type should be pinned at different paths.
> 2. Paths should be predictable, given enough information on the
> originating bpf_object (that is, adding random UID to ensure
> uniqueness is not an option).
>
> All the above should be applied to auto-pinned maps and the
> bpf_object__pin function. I am not sure if the
> bpf_object__pin_{maps,programs} should conform to those requirements
> too. Of course, all paths should be overridable similarly to the
> current implementation.
I actually think that bpf_object__pin_maps and
bpf_object__pin_programs should be removed.
bpf_object__pin()/bpf_object__unpin() and then per-map and per-program
API to control their pinning parameters should be enough to handle all
the cases.
>
> Regarding implementation, 1.c. will already be satisfied by the
> current implementation (after the program name pinning path will be
> changed, since both map names and function names are unique inside a
> single object).
That's going to change with BPF static linking. I'm thinking about
supporting static maps, i.e., maps visible within a single BPF .o
file, but still visible to outside world. At that point, each .o file
should be able to have conflicting map name, just as you'd expect to
have conflicting static variables and static functions. I haven't
thought yet all that is going to be expose to user-space, though.
> For 1.a and 1.b, I think that bpf_object__pin should produce the
> following directory layout:
>
> <obj_name>
> ├── maps
> │ └── <map_name>
> └── programs
> └── <program_name>
>
> If we decide that the requirements should apply to the specific
> bpf_object__pin_<type>s variants, then each will produce
>
> <obj_name>
> └── <type>s (i.e. maps, programs)
> └── <name>
>
> It may be better to put all pinned objects under a objects/ directory
> too, I am not sure about that.
seems a bit of an overkill, first-level directory for an object seems nice
>
> As a last point, I think that it will be nice to have a way to pin a
> bpf_object_skeleton. This will be an improvement over the current
> bpf_object__pin since skeletons keep track of attached links.
Hm.. that's the first time this comes up. You mean that all the
created bpf_links (stored inside skel->links) will be pinned in such a
case? Those links would probably go under <obj_name/links/ directory,
right? Would we then need to generate something like
my_skeleton__load_pinned(), which would be called instead of
my_skeleton__load()?
>
> There are more use cases I am not familiar with for sure, so I would
> like to hear other's opinions and comments.
Yes, absolutely, I'd like to hear some more use cases as well.
I think we need to discuss more on how to manage pinning settings for
maps (including .data, .rodata, etc) and programs. Another aspect that
is rarely discussed but is important is compatibility and
upgradeability. I.e., what if pinned map is not exactly the same as
the one you expect in your BPF code (e.g., map value size increased,
etc). This is especially important for .data, .rodata special maps, as
BPF program code will reference variables through compiled-in offsets.
For such cases we'd need to validate that all expected/used variables
are still at the same place and have the same (or compatible?) sizes.
In short, there is a lot more subtlety to pinning that meets the eye,
which is why I hope that more people will get involved in the
discussion. I personally never had a use for pinning, so for me it's
hard to judge what's important in practice. But I do see a lot of
ambiguity and potential problems with re-using BPF maps and BPF
programs :)
