On 2020-06-23T11:36:06 -0700, Andrii Nakryiko wrote:
On Tue, Jun 23, 2020 at 11:11 AM Matt Pallissard <matt@xxxxxxxxxxxxxx> wrote:
On 2020-06-23T10:58:20 -0700, Andrii Nakryiko wrote:
On Tue, Jun 23, 2020 at 9:36 AM Yonghong Song <yhs@xxxxxx> wrote:
On 6/23/20 7:54 AM, Matt Pallissard wrote:
On 2020-06-22T15:09:57 -0700, Andrii Nakryiko wrote:
On Mon, Jun 22, 2020 at 10:19 AM Matt Pallissard <matt@xxxxxxxxxxxxxx> wrote:
On 2020-06-22T09:20:03 -0700, Andrii Nakryiko wrote:
On Mon, Jun 22, 2020 at 8:01 AM Matt Pallissard <matt@xxxxxxxxxxxxxx> wrote:
On 2020-06-21T08:44:28 -0700, Matt Pallissard wrote:
On 2020-06-20T20:29:43 -0700, Andrii Nakryiko wrote:
On Sat, Jun 20, 2020 at 1:07 PM Matt Pallissard <matt@xxxxxxxxxxxxxx> wrote:
On 2020-06-20T11:11:55 -0700, Yonghong Song wrote:
On 6/20/20 9:22 AM, Matt Pallissard wrote:
New to bpf here.
I'm trying to read values out of of mm_struct. I have code like this;
unsigned long i[10] = {};
struct task_struct *t;
struct mm_rss_stat *rss;
t = (struct task_struct *)bpf_get_current_task();
BPF_CORE_READ_INTO(&rss, t, mm, rss_stat);
BPF_CORE_READ_INTO(i, rss, count);
However, all values in `i` appear to be 0 (i[MM_FILEPAGES], etc), as if no data gets copied. I'm about 100% confident that this is caused by a glaring oversight on my part.
Maybe you want to check the return value of BPF_CORE_READ_INTO.
Underlying it is using bpf_probe_read and bpf_probe_read may fail e.g., due
to major fault.
Doh, I should have known to check the return codes! Yes, it was failing. I knew I was overlooking something trivial.
I wrote exactly such piece of code a while ago. Here's part of it for
reference, I think it will be helpful:
struct task_struct *task = (struct task_struct *)bpf_get_current_task();
const struct mm_struct *mm = BPF_CORE_READ(task, mm);
if (mm) {
u64 hiwater_rss = BPF_CORE_READ(mm, hiwater_rss);
u64 file_pages = BPF_CORE_READ(mm, rss_stat.count[MM_FILEPAGES].counter);
u64 anon_pages = BPF_CORE_READ(mm, rss_stat.count[MM_ANONPAGES].counter);
u64 shmem_pages = BPF_CORE_READ(mm,
rss_stat.count[MM_SHMEMPAGES].counter);
u64 active_rss = file_pages + anon_pages + shmem_pages;
/* ... */
Thank you,
After realizing that I was referencing the struct incorrectly, I wound up with a similar block of code. However, as I started testing it against /proc/pid/smaps[,_rollup] I noticed that my numbers didn't match up. Always smaller.
I took a quick glance at fs/proc/task_mmu.c. I think I'll have to walk some sort of accounting structure.
I started to take a hard look at fs/proc/task_mmu.c. With all the locking, globals, and compile-time constants, I'm not sure that it's even possible to correctly walk `vm_area_struct` in bpf.
Yes, you can't take all those locks from BPF. But reading atomic
counters from BPF should be no problem. You might get a slightly out
of sync readings, but whatever you are doing shouldn't expect to have
100% correct values anyways, because they might change so fast after
you read them.
That was my initial thought. I didn't care to much about stale data, my only real concern was walking vm_area_struct and having memory freed. I wasn't sure if that could break the list underneath me. Although, that shouldn't be too difficult to get to the bottom of.
Not sure about vm_area_struct (where is it in the example above?), but
mm_struct won't go away, because current task won't go away, because
BPF program is running in the context of current. Similarly for
bpf_iter, bpf_iter will actually take a refcnt on tast_struct. So I
think you don't have to worry about that.
I didn't mention it explicitly in the example above. But when I originally mentioned walking an accounting structure, as procfs does, it winds up being `mm_struct->mmap,vm_[next,prev]`, with mmap being a `vm_area_struct`. But, it sounds like I should be abandoning that path and iterating over all the tasks.
If anyone has suggestions for getting memory numbers from an entire process, not just a task/thread, I'd love to hear them. If not, I'll pursue this on my own.
For this, you'd need to iterate across many tasks and aggregate their
results based on tasks's tgid. Check iter/task programs in selftests
(progs/bpf_iter_task.c, I think).
When I try to replicate some of the selftest task logic. I run into some errors when I call bpf_object__load. `libbpf: task is not found in vmlinux BTF.` I'll try matching the selftest code more closely and digging into that further.
Somehow libbpf did not prepend `task` with `bpf_iter_` prefix. Not sure
what is the exact issue. Yes, please mimic what selftests did.
It's just an artifact of how libbpf logs error in such case. It did
search for "bpf_iter_task" type, though. But Matt probably doesn't
have a recent enough kernel or didn't build it with
CONFIG_DEBUG_INFO_BTF=y and pahole 1.16+?
That shouldn't be the case, I generated vmlinux.h from my currently running machine.
I'm using an upstream kernel.
~ uname -r
5.7.2-arch1-1
Which has the BTF debug info enabled.
~ zgrep BTF= /proc/config.gz
CONFIG_DEBUG_INFO_BTF=y
I assume that it was built with the version of pahole that's in the upstream repos.
~ pacman -Ss pahole
extra/pahole 1.17-1 [installed]
Unless I've came across some odd bug, I assume that I've implemented something incorrectly.
Ok, can you show your code (BPF and user-space side) and libbpf debug logs then?
Sure. The userspace section in question is below. I don't make it past `bpf_object__load`. Same userspace code works fine for tracepoints.
struct bpf_program *prog;
struct bpf_object *obj;
char path[] = PT_BPF_OBJECT_DIR;
strcat(&path[strlen(path)], "/test.o");
libbpf_set_print(print_libbpf_log);
obj = bpf_object__open_file(path, NULL);
if (libbpf_get_error(obj))
return 1;
if(!(prog = bpf_object__find_program_by_name(obj, "dump_task")))
goto cleanup;
if (bpf_object__load(obj))
goto cleanup;
I copied the kernel code, only slightly modifying the include statements
#define bpf_iter_meta bpf_iter_meta___not_used
#define bpf_iter__task bpf_iter__task___not_used
#include <vmlinux.h>
#undef bpf_iter_meta
#undef bpf_iter__task
#include <bpf_helpers.h>
#include <bpf_tracing.h>
struct bpf_iter_meta {
struct seq_file *seq;
__u64 session_id;
__u64 seq_num;
} __attribute__((preserve_access_index));
struct bpf_iter__task {
struct bpf_iter_meta *meta;
struct task_struct *task;
} __attribute__((preserve_access_index));
SEC("iter/task")
int dump_task(struct bpf_iter__task *ctx)
{
struct seq_file *seq = ctx->meta->seq;
struct task_struct *task = ctx->task;
if (task == (void *)0) {
BPF_SEQ_PRINTF(seq, " === END ===\n");
return 0;
}
if (ctx->meta->seq_num == 0)
BPF_SEQ_PRINTF(seq, " tgid gid\n");
BPF_SEQ_PRINTF(seq, "%8d %8d\n", task->tgid, task->pid);
return 0;
}
And here is the debug output
libbpf: loading /tmp//usr/lib/pt/bpf/test.o
libbpf: section(1) .strtab, size 277, link 0, flags 0, type=3
libbpf: skip section(1) .strtab
libbpf: section(2) .text, size 0, link 0, flags 6, type=1
libbpf: skip section(2) .text
libbpf: section(3) iter/task, size 320, link 0, flags 6, type=1
libbpf: found program iter/task
libbpf: section(4) .reliter/task, size 48, link 22, flags 0, type=9
libbpf: section(5) .rodata, size 45, link 0, flags 2, type=1
libbpf: section(6) license, size 4, link 0, flags 3, type=1
libbpf: license of /tmp//usr/lib/pt/bpf/test.o is GPL
libbpf: section(7) version, size 4, link 0, flags 3, type=1
libbpf: kernel version of /tmp//usr/lib/pt/bpf/test.o is 5060b
libbpf: section(8) .debug_str, size 135270, link 0, flags 30, type=1
libbpf: skip section(8) .debug_str
libbpf: section(9) .debug_loc, size 124, link 0, flags 0, type=1
libbpf: skip section(9) .debug_loc
libbpf: section(10) .debug_abbrev, size 857, link 0, flags 0, type=1
libbpf: skip section(10) .debug_abbrev
libbpf: section(11) .debug_info, size 224491, link 0, flags 0, type=1
libbpf: skip section(11) .debug_info
libbpf: section(12) .rel.debug_info, size 160, link 22, flags 0, type=9
libbpf: skip relo .rel.debug_info(12) for section(11)
libbpf: section(13) .BTF, size 25711, link 0, flags 0, type=1
libbpf: section(14) .rel.BTF, size 80, link 22, flags 0, type=9
libbpf: skip relo .rel.BTF(14) for section(13)
libbpf: section(15) .BTF.ext, size 348, link 0, flags 0, type=1
libbpf: section(16) .rel.BTF.ext, size 288, link 22, flags 0, type=9
libbpf: skip relo .rel.BTF.ext(16) for section(15)
libbpf: section(17) .debug_frame, size 40, link 0, flags 0, type=1
libbpf: skip section(17) .debug_frame
libbpf: section(18) .rel.debug_frame, size 16, link 22, flags 0, type=9
libbpf: skip relo .rel.debug_frame(18) for section(17)
libbpf: section(19) .debug_line, size 216, link 0, flags 0, type=1
libbpf: skip section(19) .debug_line
libbpf: section(20) .rel.debug_line, size 16, link 22, flags 0, type=9
libbpf: skip relo .rel.debug_line(20) for section(19)
libbpf: section(21) .llvm_addrsig, size 6, link 22, flags 80000000, type=1879002115
libbpf: skip section(21) .llvm_addrsig
libbpf: section(22) .symtab, size 312, link 1, flags 0, type=2
libbpf: looking for externs among 13 symbols...
libbpf: collected 0 externs total
libbpf: map 'test.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 0 is "test.rodata"
libbpf: collecting relocating info for: 'iter/task'
libbpf: relo for shdr 5, symb 9, value 0, type 3, bind 0, name 0 (''), insn 7
libbpf: found data map 0 (test.rodata, sec 5, off 0) for insn 7
libbpf: relo for shdr 5, symb 9, value 0, type 3, bind 0, name 0 (''), insn 17
libbpf: found data map 0 (test.rodata, sec 5, off 0) for insn 17
libbpf: relo for shdr 5, symb 9, value 0, type 3, bind 0, name 0 (''), insn 33
libbpf: found data map 0 (test.rodata, sec 5, off 0) for insn 33
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'test.rodata': created successfully, fd=4
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: prog 'iter/task': performing 6 CO-RE offset relocs
libbpf: prog 'iter/task': relo #0: kind 0, spec is [2] bpf_iter__task + 0:0 => 0.0 @ &x[0].meta
libbpf: prog 'iter/task': relo #0: no matching targets found for [2] bpf_iter__task + 0:0
libbpf: prog 'iter/task': relo #0: substituting insn #0 w/ invalid insn
