On Tue, Mar 16, 2021 at 7:55 AM Andrii Nakryiko
<andrii.nakryiko@xxxxxxxxx> wrote:
>
> On Sun, Mar 14, 2021 at 8:40 AM Gilad Reti <gilad.reti@xxxxxxxxx> wrote:
> >
> > As libbpf is heading towards a first major release, we wanted to
> > discuss libbpf's object pinning strategy.
> >
> > bpf object pinning has a couple of use cases (feel free to add, there
> > are more for sure):
> > 1. Sharing specific bpf objects between different processes (for
> > example, one process loads a bpf skeleton, another one interacts with
> > it using various bpf maps (for example, for changing configurations
> > (i.e. dynamic networking rules etc))
> > 2. Preventing bpf objects from destruction upon owning process exit
> > (i.e. to prevent bpf progs detach upon userspace program crash)
> >
> > Regarding the first use case, for most cases manually setting the pin
> > path (both in the loading process and in other processes) will
> > probably be the best. In such cases, no redesign is required here.
> >
> > For the second one, something like the bpf_object__pin will be more
> > appropriate (to allow a complete reuse of the bpf objects). For that
> > use case, some sensible requirements we can consider are:
> >
> > 1. Paths should be unique:
> > a. at the bpf_object level (that is, same pinnable objects that
> > belong to different bpf_object s should be pinned at different paths).
> > b. in the same bpf_object, between different pinnable object types
> > (i.e. a map and a prog) should always be pinned at different paths.
> > c. different objects, belonging to the same bpf_object and of the
> > same type should be pinned at different paths.
> > 2. Paths should be predictable, given enough information on the
> > originating bpf_object (that is, adding random UID to ensure
> > uniqueness is not an option).
> >
> > All the above should be applied to auto-pinned maps and the
> > bpf_object__pin function. I am not sure if the
> > bpf_object__pin_{maps,programs} should conform to those requirements
> > too. Of course, all paths should be overridable similarly to the
> > current implementation.
>
> I actually think that bpf_object__pin_maps and
> bpf_object__pin_programs should be removed.
> bpf_object__pin()/bpf_object__unpin() and then per-map and per-program
> API to control their pinning parameters should be enough to handle all
> the cases.
>
Sure. I personally couldn't find any usecase for exposing them, and if
there is no such one then I am totally okay with that.
> >
> > Regarding implementation, 1.c. will already be satisfied by the
> > current implementation (after the program name pinning path will be
> > changed, since both map names and function names are unique inside a
> > single object).
>
> That's going to change with BPF static linking. I'm thinking about
> supporting static maps, i.e., maps visible within a single BPF .o
> file, but still visible to outside world. At that point, each .o file
> should be able to have conflicting map name, just as you'd expect to
> have conflicting static variables and static functions. I haven't
> thought yet all that is going to be expose to user-space, though.
>
> > For 1.a and 1.b, I think that bpf_object__pin should produce the
> > following directory layout:
> >
> > <obj_name>
> > ├── maps
> > │ └── <map_name>
> > └── programs
> > └── <program_name>
> >
> > If we decide that the requirements should apply to the specific
> > bpf_object__pin_<type>s variants, then each will produce
> >
> > <obj_name>
> > └── <type>s (i.e. maps, programs)
> > └── <name>
> >
> > It may be better to put all pinned objects under a objects/ directory
> > too, I am not sure about that.
>
> seems a bit of an overkill, first-level directory for an object seems nice
>
> >
> > As a last point, I think that it will be nice to have a way to pin a
> > bpf_object_skeleton. This will be an improvement over the current
> > bpf_object__pin since skeletons keep track of attached links.
>
> Hm.. that's the first time this comes up. You mean that all the
> created bpf_links (stored inside skel->links) will be pinned in such a
> case? Those links would probably go under <obj_name/links/ directory,
> right? Would we then need to generate something like
> my_skeleton__load_pinned(), which would be called instead of
> my_skeleton__load()?
>
Yes, something like that.
> >
> > There are more use cases I am not familiar with for sure, so I would
> > like to hear other's opinions and comments.
>
> Yes, absolutely, I'd like to hear some more use cases as well.
>
> I think we need to discuss more on how to manage pinning settings for
> maps (including .data, .rodata, etc) and programs. Another aspect that
> is rarely discussed but is important is compatibility and
> upgradeability. I.e., what if pinned map is not exactly the same as
> the one you expect in your BPF code (e.g., map value size increased,
> etc). This is especially important for .data, .rodata special maps, as
> BPF program code will reference variables through compiled-in offsets.
> For such cases we'd need to validate that all expected/used variables
> are still at the same place and have the same (or compatible?) sizes.
>
> In short, there is a lot more subtlety to pinning that meets the eye,
> which is why I hope that more people will get involved in the
> discussion. I personally never had a use for pinning, so for me it's
> hard to judge what's important in practice. But I do see a lot of
> ambiguity and potential problems with re-using BPF maps and BPF
> programs :)
