>> I don't have a 3.10 kernel installed. Below you attach a probe to
>> nfs_readpages:46, which is the 46th line after the function entry
>> point. Can you do an objdump of the nfs.ko to see where
>> nfs_symlink_filter is? As it may just be the symbol that corresponds to
>> the location perf found reading the dwarf information.
a-b06-u47 % objdump -t ~/foo | grep -A2 -B2 nfs_symlink_filler
0000000000000000 l df *ABS* 0000000000000000 symlink.c
00000000000151c0 l F .text 0000000000000095 nfs_follow_link
0000000000015260 l F .text 000000000000005b nfs_symlink_filler
0000000000000000 l df *ABS* 0000000000000000 unlink.c
00000000000152c0 l F .text 000000000000002d nfs_unlink_prepare
> Could you list the probe event by perf-probe ("perf probe -l")
> instead of dumping kprobe_events?
> If it is on an incorrect address, the listed line should be wrong.
>
> And could you also check the perf version? ("perf -v")
> Old perf might have that problem.
Here’s another example:
% sudo perf probe -v -m nfs -n -a 'nfs_writeback_done:18 inode=inode bytes=hdr->res.count’
probe-definition(0): nfs_writeback_done:18 inode=inode bytes=hdr->res.count
symbol:nfs_writeback_done file:(null) line:18 offset:0 return:0 lazy:(null)
parsing arg: inode=inode into name:inode inode
parsing arg: bytes=hdr->res.count into name:bytes hdr, res(1), count(0)
2 arguments
No kprobe blacklist support, ignored
Failed to get build-id from nfs.
Cache open error: -1
Open Debuginfo file: /lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko.xz
Try to find probe point from debuginfo.
Matched function: nfs_writeback_done [19cc82]
Probe point found: nfs_writeback_done+91
Searching 'inode' variable in context.
Converting variable inode into trace event.
inode type is (null).
Searching 'hdr' variable in context.
Converting variable hdr into trace event.
converting res in hdr
converting count in res
count type is unsigned int.
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing//kprobe_events write=1
Writing event: p:probe/nfs_writeback_done nfs:nfs_writeback_done+182 inode=%r12:u64 bytes=+736(%bx):u32
Added new event:
probe:nfs_writeback_done (on nfs_writeback_done:18 in nfs with inode=inode bytes=hdr->res.count)
You can now use it in all perf tools, such as:
perf record -e probe:nfs_writeback_done -aR sleep 1
I note that in the output I’m seeing two different offsets (+91 vs +182); does this point to a perf tool problem?
"Probe point found: nfs_writeback_done+91”
and
Writing event: p:probe/nfs_writeback_done nfs:nfs_writeback_done+182 inode=%r12:u64 bytes=+736(%bx):u32
The perf version (on RHEL) is perf version 3.10.0-693.2.2.el7.x86_64.debug
Thanks!
Nick
> On Mar 17, 2018, at 12:50 AM, Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:
>
> Hi Nick,
>
> On Thu, 15 Mar 2018 17:33:31 -0400
> Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
>
>> On Thu, 15 Mar 2018 11:05:01 -0700
>> Nick Kirsch <nick@xxxxxxxxxx> wrote:
>>
>>> Hi!
>>>
>>> I’m having some issues with symbol resolution; namely, perf probe
>>> seems to think it’s happily attaching to the symbol
>>> (nfs_readpages:362) I’m looking for and yet it shows up as a
>>> different symbol (nfs_symlink_filter+36).
>>
>
>
>
> Thanks,
>
>>
>> -- Steve
>>
>>
>>>
>>> I’d love some help!
>>>
>>> # uname -a
>>>
>>> Linux a-b06-u47 3.10.0-693.2.2.el7.x86_64 #1 SMP Sat Sep 9 03:55:24
>>> EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> # perf probe -v
>>> -m /usr/lib/debug/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko.debug
>>> -a 'nfs_readpages:46 pgm->pg_bytes_written' probe-definition(0):
>>> nfs_readpages:46 pgm->pg_bytes_written symbol:nfs_readpages
>>> file:(null) line:46 offset:0 return:0 lazy:(null) parsing arg:
>>> pgm->pg_bytes_written into pgm, pg_bytes_written(1) 1 arguments No
>>> kprobe blacklist support, ignored Open Debuginfo
>>> file: /usr/lib/debug/usr/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko.debug
>>> Try to find probe point from debuginfo. Matched function:
>>> nfs_readpages [143625] Probe point found: nfs_readpages+362
>>> Searching 'pgm' variable in context.
>>> Converting variable pgm into trace event.
>>> converting pg_bytes_written in pgm
>>> pg_bytes_written type is long unsigned int.
>>> Found 1 probe_trace_events.
>>> Opening /sys/kernel/debug/tracing//kprobe_events write=1
>>> Writing event:
>>> p:probe/nfs_readpages /usr/lib/debug/usr/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko.debug:nfs_symlink_filler+36
>>> pg_bytes_written=+16(%ax):u64 Added new event: probe:nfs_readpages
>>> (on nfs_readpages:46
>>> in /usr/lib/debug/usr/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko.debug
>>> with pg_bytes_written=pgm->pg_bytes_written)
>>>
>>> [root@a-b06-u47 tracing]# cat kprobe_events
>>> p:probe/nfs_readpages /usr/lib/debug/usr/lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko.debug:nfs_symlink_filler+36
>>> pg_bytes_written=+16(%ax):u64
>>>
>>>
>>> Thanks!
>>>
>>> Nick--
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-trace-users" in the body of a message to
>>> majordomo@xxxxxxxxxxxxxxx More majordomo info at
>>> http://vger.kernel.org/majordomo-info.html
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> --
> Masami Hiramatsu <mhiramat@xxxxxxxxxx>
--
To unsubscribe from this list: send the line "unsubscribe linux-trace-users" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
|