On Mon, May 06, 2024 at 05:07:54PM GMT, Arnaldo Carvalho de Melo wrote:
> On Tue, Apr 30, 2024 at 05:27:24PM -0600, Daniel Xu wrote:
> > On Tue, Apr 30, 2024 at 05:42:14PM GMT, Arnaldo Carvalho de Melo wrote:
> > > On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> > > > This commit teaches pahole to parse symbols in .BTF_ids section in
> > > > vmlinux and discover exported kfuncs. Pahole then takes the list of
> > > > kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> > > >
> > > > Example of encoding:
> > > >
> > > > $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
> > > > 121
> > > >
> > > > $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
> > > > [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
> > > > [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> > > >
> > > > This enables downstream users and tools to dynamically discover which
> > > > kfuncs are available on a system by parsing vmlinux or module BTF, both
> > > > available in /sys/kernel/btf.
> > > >
> > > > This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.
> > >
> > > I'm trying this but:
> > >
> > > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > btf_encoder__tag_kfuncs(cgroup_rstat_updated): found=0
> > > btf_encoder__tag_kfuncs(cgroup_rstat_flush): found=0
> > > btf_encoder__tag_kfuncs(security_file_permission): found=0
> > > btf_encoder__tag_kfuncs(security_inode_getattr): found=0
> > > btf_encoder__tag_kfuncs(security_file_open): found=0
> > > btf_encoder__tag_kfuncs(security_path_truncate): found=0
> > > btf_encoder__tag_kfuncs(vfs_truncate): found=0
> > > btf_encoder__tag_kfuncs(vfs_fallocate): found=0
> > > btf_encoder__tag_kfuncs(dentry_open): found=0
> > > btf_encoder__tag_kfuncs(vfs_getattr): found=0
> > > btf_encoder__tag_kfuncs(filp_close): found=0
> > > btf_encoder__tag_kfuncs(bpf_lookup_user_key): found=0
> > > btf_encoder__tag_kfuncs(bpf_lookup_system_key): found=0
> > > btf_encoder__tag_kfuncs(bpf_key_put): found=0
> > > btf_encoder__tag_kfuncs(bpf_verify_pkcs7_signature): found=0
> > > btf_encoder__tag_kfuncs(bpf_obj_new_impl): found=0
> > > <SNIP all with found=0>
> > >
> > > With:
> > >
> > > ⬢[acme@toolbox pahole]$ git diff -U16
> > > diff --git a/btf_encoder.c b/btf_encoder.c
> > > index c2df2bc7a374447b..27a16d6564381b60 100644
> > > --- a/btf_encoder.c
> > > +++ b/btf_encoder.c
> > > @@ -1689,32 +1689,35 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
> > > func = get_func_name(name);
> > > if (!func)
> > > continue;
> > >
> > > /* Check if function belongs to a kfunc set */
> > > ranges = gobuffer__entries(&btf_kfunc_ranges);
> > > ranges_cnt = gobuffer__nr_entries(&btf_kfunc_ranges);
> > > found = false;
> > > for (j = 0; j < ranges_cnt; j++) {
> > > size_t addr = sym.st_value;
> > >
> > > if (ranges[j].start <= addr && addr < ranges[j].end) {
> > > found = true;
> > > break;
> > > }
> > > }
> > > +
> > > + printf("%s(%s): found=%d\n", __func__, func, found);
> > > +
> > > if (!found) {
> > > free(func);
> > > continue;
> > > }
> > >
> > > err = btf_encoder__tag_kfunc(encoder, &btf_funcs, func);
> > > if (err) {
> > > fprintf(stderr, "%s: failed to tag kfunc '%s'\n", __func__, func);
> > > free(func);
> > > goto out;
> > > }
> > > free(func);
> > > }
> > >
> > > err = 0;
> > > out:
> > >
> > > --------------
> > >
> > > The vmlinux I'm testing on has the kfuncs, etc, as we can see with:
> > >
> > > ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | wc -l
> > > 517
> > > ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | tail
> > > 97887: ffffffff83266bfc 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__cubictcp_cong_avoid__805493
> > > 97888: ffffffff83266c04 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__cubictcp_state__806494
> > > 97889: ffffffff83266c0c 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__cubictcp_cwnd_event__807495
> > > 97890: ffffffff83266c14 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__cubictcp_acked__808496
> > > 98068: ffffffff83266c24 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__tcp_reno_ssthresh__773199
> > > 98069: ffffffff83266c2c 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__tcp_reno_cong_avoid__774200
> > > 98070: ffffffff83266c34 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__tcp_reno_undo_cwnd__775201
> > > 98071: ffffffff83266c3c 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__tcp_slow_start__776202
> > > 98072: ffffffff83266c44 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__tcp_cong_avoid_ai__777203
> > > 101522: ffffffff83266c5c 4 OBJECT LOCAL DEFAULT 15 __BTF_ID__func__update_socket_protocol__80024
> > > ⬢[acme@toolbox pahole]$
> > >
> > >
> > > So that btf_encoder__tag_kfuncs() isn't finding any?
> > >
> > > $ pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > btf_encoder__tag_kfuncs(vmlinux)
> > >
> > > Yeah, getting the source filename, the right one.
> > >
> > > Then is_sym_kfunc_set() never returns true... But:
> > >
> > > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > >
> > > real 0m5.586s
> > > user 0m29.707s
> > > sys 0m2.160s
> > > ⬢[acme@toolbox pahole]$
> > >
> > > And then:
> > >
> > > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> >
> > set->flags=0 here is odd. I'd expect at least some of those to be
> > non-zero. Can you check if your tree has
> > https://github.com/torvalds/linux/commit/6f3189f38a3e995232e028a4c341164c4aca1b20
> > ?
>
> ⬢[acme@toolbox linux]$ git tag --contains 6f3189f38a3e995232e028a4c341164c4aca1b20
> v6.9-rc1
> v6.9-rc2
> v6.9-rc3
> v6.9-rc4
> v6.9-rc5
> v6.9-rc6
> v6.9-rc7
> ⬢[acme@toolbox linux]$ git log --oneline -1
> dd5a440a31fae6e4 (HEAD, tag: v6.9-rc7, torvalds/master) Linux 6.9-rc7
> ⬢[acme@toolbox linux]$
>
> So now with a just built upstream kernel I get the output below, do you
> have patches for other tools to consume this? Or does, say, bpftrace
> already handles such decl tags, etc?
Yep, I have v3 of https://lore.kernel.org/bpf/cover.1707080349.git.dxu@xxxxxxxxx/
ready to send. The step after that bpftool patch is merged is
teach/simplify the bpf selftests. Long term once all the changes
propagate, it'll make bpf programmers workflows more efficient too.
bpftrace will probably make use of these tags in the future. We are
still working on our kfunc story (not supported yet).
[..]
> [135552] DECL_TAG 'bpf_kfunc' type_id=33061 component_idx=-1
> [135553] DECL_TAG 'bpf_kfunc' type_id=134975 component_idx=-1
> [135554] DECL_TAG 'bpf_kfunc' type_id=134971 component_idx=-1
> [135555] DECL_TAG 'bpf_kfunc' type_id=134972 component_idx=-1
> [135556] DECL_TAG 'bpf_kfunc' type_id=134970 component_idx=-1
> [135557] DECL_TAG 'bpf_kfunc' type_id=134974 component_idx=-1
> [135558] DECL_TAG 'bpf_kfunc' type_id=134969 component_idx=-1
> [135559] DECL_TAG 'bpf_kfunc' type_id=53825 component_idx=-1
> [135560] DECL_TAG 'bpf_kfunc' type_id=53827 component_idx=-1
> [135561] DECL_TAG 'bpf_kfunc' type_id=53824 component_idx=-1
> [135562] DECL_TAG 'bpf_kfunc' type_id=53831 component_idx=-1
> [135563] DECL_TAG 'bpf_kfunc' type_id=53829 component_idx=-1
> [135564] DECL_TAG 'bpf_kfunc' type_id=21317 component_idx=-1
> [135565] DECL_TAG 'bpf_kfunc' type_id=21315 component_idx=-1
> ⬢[acme@toolbox pahole]$
Glad it works!
Thanks,
Daniel
