On 31/01/2024 17:06, Daniel Xu wrote:
> Hi Alan,
>
> On Wed, Jan 31, 2024 at 04:20:01PM +0000, Alan Maguire wrote:
>> Adding userspace tracepoints in other languages like python and
>> go is a very useful for observability. libstapsdt [1]
>> and language bindings like python-stapsdt [2] that rely on it
>> use a clever scheme of emulating static (USDT) userspace tracepoints
>> at runtime. This involves (as I understand it):
>>
>> - fabricating a shared library
>> - annotating it with ELF notes that describe its tracepoints
>> - dlopen()ing it and calling the appropriate probe fire function
>> to trigger probe firing.
>>
>> bcc already supports this mechanism (the examples in [2] use
>> bcc to list/trigger the tracepoints), so it seems like it
>> would be a good candidate for adding support to libbpf.
>>
>> However, before doing that, it's worth considering if there
>> are simpler ways to support runtime probe firing. This
>> small series demonstrates a simple method based on USDT
>> probes added to libbpf itself.
>>
>> The suggested solution comprises 3 parts
>>
>> 1. functions to fire dynamic probes are added to libbpf itself
>> bpf_urdt__probeN(), where N is the number of probe arguemnts.
>> A sample usage would be
>> bpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);
>>
>> Under the hood these correspond to USDT probes with an
>> additional argument for uniquely identifying the probe
>> (a hash of provider/probe name).
>>
>> 2. we attach to the appropriate USDT probe for the specified
>> number of arguments urdt/probe0 for none, urdt/probe1 for
>> 1, etc. We utilize the high-order 32 bits of the attach
>> cookie to store the hash of the provider/probe name.
>>
>> 3. when urdt/probeN fires, the BPF_URDT() macro (which
>> is similar to BPF_USDT()) checks if the hash passed
>> in (identifying provider/probe) matches the attach
>> cookie high-order 32 bits; if not it must be a firing
>> for a different dynamic probe and we exit early.
>>
>> Auto-attach support is also added, for example the following
>> would add a dynamic probe for provider:myprobe:
>>
>> SEC("udrt/libbpf.so:2:myprovider:myprobe")
>> int BPF_URDT(myprobe, int arg1, char *arg2)
>> {
>> ...
>> }
>>
>> (Note the "2" above specifies the number of arguments to
>> the probe, otherwise it is identical to USDT).
>>
>> The above program can then be triggered by a call to
>>
>> BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");
>>
>> The useful thing about this is that by attaching to
>> libbpf.so (and firing probes using that library) we
>> can get system-wide dynamic probe firing. It is also
>> easy to fire a dynamic probe - no setup is required.
>>
>> More examples of auto and manual attach can be found in
>> the selftests (patch 2).
>>
>> If this approach appears to be worth pursing, we could
>> also look at adding support to libstapsdt for it.
>
> This is quite interesting, thanks for the RFC. I hope to take a closer
> look at it this week.
>
> At a high level, it looks like you're basically defining a scheme for
> well-known USDT probes, right? Since, not all languages enjoy linking
> against C (looking at you golang...), perhaps it would make sense to
> codify the scheme in a "spec". Probably just located in Documentation/
> or something. That way there can be independent implementations.
>
That's a great idea. If tracers or libraries like libstapsdt wanted
to define their own probe firings, we could add option fields to
the urdt attach function to specify the USDT provider/probe names
for attachment for runtime probe emulation. If another entity was
specifying USDT probes like this, it would also need to pass through a
hash value, so that triple could potentially be added to the urdt attach
options; doing that would give other entities like libstapsdt a free
hand to define their own trigger mechanisms, and not have to link to
libbpf. That said, the benefits of a common scheme are that libbpf does
not have to figure out which USDT probes to use in advance, so having
defaults that are documented makes sense. It's especially useful for
auto-attach by SEC() name.
> This is something that would be nice to support in bpftrace as well. I'm
> sure other tracers would probably find use as well.
>
Yeah, I was aiming for as simple as possible to make it easy for tracers
to adopt. The aim is to try and facilitate wider support for runtime
probes, so I'm curious if other folks have run into issues in that area,
or have suggestions.
Thanks!
Alan
> Thanks,
> Daniel
