Hi Evgenii,
On Thu, 18 Feb 2021 20:09:17 +0300
Evgenii Shatokhin <eshatokhin@xxxxxxxxxxxxx> wrote:
> Hi,
>
> It seems, 'perf probe' can only see functions from .text section in the
> kernel modules, but not from .text.unlikely or other .text.* sections.
>
> For example, with kernel 5.11 and nf_conntrack.ko with debug info, 'perf
> probe' succeeds for nf_conntrack_attach() from .text and fails for
> nf_ct_resolve_clash() from .text.unlikely:
Thanks for reporting it!
>
> ------------
> # perf probe -v -m nf_conntrack nf_ct_resolve_clash
> probe-definition(0): nf_ct_resolve_clash
> symbol:nf_ct_resolve_clash file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Failed to get build-id from nf_conntrack.
> Cache open error: -1
> Open Debuginfo file:
> /lib/modules/5.11.0-test01/kernel/net/netfilter/nf_conntrack.ko
> Try to find probe point from debuginfo.
> Matched function: nf_ct_resolve_clash [33616]
> Probe point found: nf_ct_resolve_clash+0
> Found 1 probe_trace_events.
> Post processing failed or all events are skipped. (-2)
> Probe point 'nf_ct_resolve_clash' not found.
> Error: Failed to add events. Reason: No such file or directory (Code: -2)
The above log shows, an error occured while post_process_probe_trace_events(),
and the error code is -ENOENT (-2).
----
pr_debug("Found %d probe_trace_events.\n", ntevs);
ret = post_process_probe_trace_events(pev, *tevs, ntevs,
pev->target, pev->uprobes, dinfo);
if (ret < 0 || ret == ntevs) {
pr_debug("Post processing failed or all events are skipped. (%d)\n", ret);
----
In that function, map__find_symbol() failure will return -ENOENT.
----
/* Adjust symbol name and address */
static int post_process_probe_trace_point(struct probe_trace_point *tp,
struct map *map, unsigned long offs)
{
struct symbol *sym;
u64 addr = tp->address - offs;
sym = map__find_symbol(map, addr);
if (!sym)
return -ENOENT;
----
So it seems "map" may not load the symbol out of ".text".
This need to be fixed, since the map is widely used in the perf.
Anyway, since this is on a module, so even if it can not find the symbol
from map (or failed to load a map), it can fail back to the original symbol.
Let me fix that.
> # perf probe -v -m nf_conntrack nf_conntrack_attach
> probe-definition(0): nf_conntrack_attach
> symbol:nf_conntrack_attach file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Failed to get build-id from nf_conntrack.
> Cache open error: -1
> Open Debuginfo file:
> /lib/modules/5.11.0-test01/kernel/net/netfilter/nf_conntrack.ko
> Try to find probe point from debuginfo.
> Matched function: nf_conntrack_attach [2c8c3]
> Probe point found: nf_conntrack_attach+0
> Found 1 probe_trace_events.
> Opening /sys/kernel/tracing//kprobe_events write=1
> Opening /sys/kernel/tracing//README write=0
> Writing event: p:probe/nf_conntrack_attach
> nf_conntrack:nf_conntrack_attach+0
> Added new event:
> probe:nf_conntrack_attach (on nf_conntrack_attach in nf_conntrack)
> ------------
>
> Is there a way to allow probing of functions in .text.<something> ?
I need to check how machine__kernel_maps() generated maps cut down .text.unlikely.
Arnaldo, I thought the maps in machine__kernel_maps() are generated from
kallsyms (doesn't check .text) right?
>
> Of course, one could place probes using absolute addresses of the
> functions but that would be less convenient.
>
> This also affects many livepatch modules where the kernel code can be
> compiled with -ffunction-sections and each function may end up in a
> separate section .text.<function_name>. 'perf probe' cannot be used
> there, except with the absolute addresses.
>
> Moreover, if FGKASLR patches are merged
> (https://lwn.net/Articles/832434/) and the kernel is built with FGKASLR
> enabled, -ffunction-sections will be used too. 'perf probe' will be
> unable to see the kernel functions then.
Hmm, if the FGKASLAR really randomizes the symbol address, perf-probe
should give up "_text-relative" probe for that kernel, and must fallback
to the "symbol-based" probe. (Are there any way to check the FGKASLR is on?)
The problem of "symbol-based" probe is that local (static) symbols
may share a same name sometimes. In that case, it can not find correct
symbol. (Maybe I can find a candidate from its size.)
Anyway, sometimes the security and usability are trade-off.
Thank you,
--
Masami Hiramatsu <mhiramat@xxxxxxxxxx>
