On Tue, Feb 6, 2024 at 1:49 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
>
> On 02/02/2024 21:39, Andrii Nakryiko wrote:
> > On Wed, Jan 31, 2024 at 8:20 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
> >>
> >> Adding userspace tracepoints in other languages like python and
> >> go is a very useful for observability. libstapsdt [1]
> >> and language bindings like python-stapsdt [2] that rely on it
> >> use a clever scheme of emulating static (USDT) userspace tracepoints
> >> at runtime. This involves (as I understand it):
> >>
> >> - fabricating a shared library
> >> - annotating it with ELF notes that describe its tracepoints
> >> - dlopen()ing it and calling the appropriate probe fire function
> >> to trigger probe firing.
> >>
> >> bcc already supports this mechanism (the examples in [2] use
> >> bcc to list/trigger the tracepoints), so it seems like it
> >> would be a good candidate for adding support to libbpf.
> >>
> >> However, before doing that, it's worth considering if there
> >> are simpler ways to support runtime probe firing. This
> >> small series demonstrates a simple method based on USDT
> >> probes added to libbpf itself.
> >>
> >> The suggested solution comprises 3 parts
> >>
> >> 1. functions to fire dynamic probes are added to libbpf itself
> >> bpf_urdt__probeN(), where N is the number of probe arguemnts.
> >> A sample usage would be
> >> bpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);
> >>
> >> Under the hood these correspond to USDT probes with an
> >> additional argument for uniquely identifying the probe
> >> (a hash of provider/probe name).
> >>
> >> 2. we attach to the appropriate USDT probe for the specified
> >> number of arguments urdt/probe0 for none, urdt/probe1 for
> >> 1, etc. We utilize the high-order 32 bits of the attach
> >> cookie to store the hash of the provider/probe name.
> >>
> >> 3. when urdt/probeN fires, the BPF_URDT() macro (which
> >> is similar to BPF_USDT()) checks if the hash passed
> >> in (identifying provider/probe) matches the attach
> >> cookie high-order 32 bits; if not it must be a firing
> >> for a different dynamic probe and we exit early.
> >
> > I'm sorry Alan, but I don't see this being added to libbpf. This is
> > nothing else than USDT with a bunch of extra conventions bolted on.
> > And those conventions might not work for many environments. It is
> > completely arbitrary that libbpf is a) assumed to be a dynamic library
> > and b) provides USDT hooks that will be triggered. Just because it
> > will be libbpf that will be used to trace those USDT hooks doesn't
> > mean that libbpf has to define those hooks.
>
> Right - that came up with the discussion with Daniel also. Adding
> probes in libbpf was just a means of providing a method of last resort
> for runtime probe firing, it's not strictly necessary.
>
Ok, glad we agree on not putting this into libbpf.
> > Just because libbpf can
> > trace USDTs it doesn't mean that libbpf should provide those
> > STAP_PROBEx() macros to define and trigger USDTs within some
> > application. Applications that define USDTs and applications that
> > attach to those USDTs are completely separate and independent. Same
> > here, there might be an overlap in some cases, but conceptually it's
> > two separate sides of the solution.
> >
>
> I think the point is though that USDT got its start by establishing a
> shared set of conventions between the to-be-traced side and the tracer,
> and built upon existing uprobe support to make that work. Is there a
> similar approach that we could apply for dynamic probes? libbpf isn't
> necessarily the right vehicle for establishing those conventions and I'm
> far from wedded to the specifics of this approach, but I do think it's a
> question we should explore a bit.
I'm a bit skeptical about "standardizing" this approach so early.
Let's see how and whether this gets used in practice widely enough,
before adding SEC("urdt") as a standard feature into libbpf.
>
> > Overall, this is definitely a useful overall approach, to have a
> > single system-wide .so library that can be attached to trace some
> > USDTs, and we've explored this approach internally at Meta as well.
> > But I don't believe it should be part of libbpf. From libbpf's
> > standpoint it's just a standard USDT probe to attach to.
> >
> >
>
> For now we can pursue the approach of adding a static probe - triggered
> when a dynamic probe firing is requested - to libstapsdt, and this would
> give us libbpf support via USDT tracing of libstapsdt.
Exactly. And I suspect big companies like Meta, Google and whatnot
might have their own small system-wide shared library, developed and
maintained internally, deployed using whatever deployment strategy
they see fit, with their own USDT name and convention about naming and
namespacing these dynamic USDTs, etc, etc.
As I said, the idea is sound and neat, I'm just not sure that assuming
libstapsdt (or whatever other predefined library) is going to work in
practice.
If anything, maybe making the kernel provide something like this as a
standard functionality is the right way to go. Something along the
VDSO lines, which doesn't trigger a context switch, but can be
centralized through the kernel? Have you considered such options?
>
> >>
> >> Auto-attach support is also added, for example the following
> >> would add a dynamic probe for provider:myprobe:
> >>
> >> SEC("udrt/libbpf.so:2:myprovider:myprobe")
> >> int BPF_URDT(myprobe, int arg1, char *arg2)
> >> {
> >> ...
> >> }
> >>
> >> (Note the "2" above specifies the number of arguments to
> >> the probe, otherwise it is identical to USDT).
> >>
> >> The above program can then be triggered by a call to
> >>
> >> BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");
> >>
> >> The useful thing about this is that by attaching to
> >> libbpf.so (and firing probes using that library) we
> >> can get system-wide dynamic probe firing. It is also
> >> easy to fire a dynamic probe - no setup is required.
> >>
> >> More examples of auto and manual attach can be found in
> >> the selftests (patch 2).
> >>
> >> If this approach appears to be worth pursing, we could
> >> also look at adding support to libstapsdt for it.
> >>
> >> Alan Maguire (2):
> >> libbpf: add support for Userspace Runtime Dynamic Tracing (URDT)
> >> selftests/bpf: add tests for Userspace Runtime Defined Tracepoints
> >> (URDT)
> >>
> >> tools/lib/bpf/Build | 2 +-
> >> tools/lib/bpf/Makefile | 2 +-
> >> tools/lib/bpf/libbpf.c | 94 ++++++++++
> >> tools/lib/bpf/libbpf.h | 94 ++++++++++
> >> tools/lib/bpf/libbpf.map | 13 ++
> >> tools/lib/bpf/libbpf_internal.h | 2 +
> >> tools/lib/bpf/urdt.bpf.h | 103 +++++++++++
> >> tools/lib/bpf/urdt.c | 145 +++++++++++++++
> >> tools/testing/selftests/bpf/Makefile | 2 +-
> >> tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++
> >> tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++
> >> .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++
> >> 12 files changed, 786 insertions(+), 3 deletions(-)
> >> create mode 100644 tools/lib/bpf/urdt.bpf.h
> >> create mode 100644 tools/lib/bpf/urdt.c
> >> create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c
> >> create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c
> >> create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c
> >>
> >> --
> >> 2.39.3
> >>
> >
