Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> [Fri, 2020-04-10 14:52 -0700]:
> On Fri, Apr 10, 2020 at 2:14 PM Alexei Starovoitov
> <alexei.starovoitov@xxxxxxxxx> wrote:
> >
> > On Fri, Apr 10, 2020 at 01:39:03PM -0700, Andrii Nakryiko wrote:
> > > On Fri, Apr 10, 2020 at 12:54 PM Andrey Ignatov <rdna@xxxxxx> wrote:
> > > >
> > > > Initially BPF_CGROUP_INET_EGRESS hook didn't require specifying
> > > > expected_attach_type at loading time, but commit
> > > >
> > > > 5cf1e9145630 ("bpf: cgroup inet skb programs can return 0 to 3")
> > > >
> > > > changed it so that expected_attach_type must be specified if program can
> > > > return either 2 or 3 (before it was either 0 or 1) to communicate
> > > > congestion notification to caller.
> > > >
> > > > At the same time loading w/o expected_attach_type is still supported for
> > > > backward compatibility if program retval is in tnum_range(0, 1).
> > > >
> > > > Though libbpf currently supports guessing prog/attach/expected_attach
> > > > types only for "old" mode (retval in [0; 1]). And if cgroup_skb egress
> > > > program stars returning e.g. 2 (corresponds to NET_XMIT_CN), then
> > > > guessing breaks and, e.g. bpftool can't load an object with such a
> > > > program anymore:
> > > >
> > > > # bpftool prog loadall tools/testing/selftests/bpf/test_skb.o /sys/fs/bpf/test_skb
> > > > libbpf: load bpf program failed: Invalid argument
> > > > libbpf: -- BEGIN DUMP LOG ---
> > > > libbpf:
> > > > ; return tc_prog(skb) == TC_ACT_OK ? 1 : 2 /* NET_XMIT_CN */;
> > > > 0: (85) call pc+5
> > > >
> > > > ... skip ...
> > > >
> > > > from 87 to 1: R0_w=invP2 R10=fp0
> > > > ; return tc_prog(skb) == TC_ACT_OK ? 1 : 2 /* NET_XMIT_CN */;
> > > > 1: (bc) w1 = w0
> > > > 2: (b4) w0 = 1
> > > > 3: (16) if w1 == 0x0 goto pc+1
> > > > 4: (b4) w0 = 2
> > > > ; return tc_prog(skb) == TC_ACT_OK ? 1 : 2 /* NET_XMIT_CN */;
> > > > 5: (95) exit
> > > > At program exit the register R0 has value (0x2; 0x0) should have been in (0x0; 0x1)
> > > > processed 96 insns (limit 1000000) max_states_per_insn 1 total_states 10 peak_states 10 mark_read 2
> > > >
> > > > libbpf: -- END LOG --
> > > > libbpf: failed to load program 'cgroup_skb/egress'
> > > > libbpf: failed to load object 'tools/testing/selftests/bpf/test_skb.o'
> > > > Error: failed to load object file
> > > >
> > > > Fix it by introducing another entry in libbpf section_defs that makes the load
> > > > happens with expected_attach_type: cgroup_skb/egress/expected
> > > >
> > > > That name may not be ideal, but I don't have a better option.
> > >
> > > That's a really bad name :) But maybe instead of having another
> > > section_def, turn existing section def into the one that does specify
> > > expected_attach_type? Seems like kernels accept expected_attach_type
> > > for a while now, so it might be ok backwards compatibility-wise?
> > > Otherwise, we can teach libbpf to retry program load without expected
> > > attach type for cgroup_skb/egress?
> > >
> > > >
> > > > Strictly speaking this is not a fix but rather a missing feature, that's
> > > > why there is no Fixes tag. But it still seems to be a good idea to merge
> > > > it to stable tree to fix loading programs that use a feature available
> > > > for almost a year.
> > > >
> > > > Signed-off-by: Andrey Ignatov <rdna@xxxxxx>
> > > > ---
> > > > tools/lib/bpf/libbpf.c | 2 ++
> > > > 1 file changed, 2 insertions(+)
> > > >
> > > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > > > index ff9174282a8c..c909352f894d 100644
> > > > --- a/tools/lib/bpf/libbpf.c
> > > > +++ b/tools/lib/bpf/libbpf.c
> > > > @@ -6330,6 +6330,8 @@ static const struct bpf_sec_def section_defs[] = {
> > > > BPF_PROG_SEC("lwt_seg6local", BPF_PROG_TYPE_LWT_SEG6LOCAL),
> > > > BPF_APROG_SEC("cgroup_skb/ingress", BPF_PROG_TYPE_CGROUP_SKB,
> > > > BPF_CGROUP_INET_INGRESS),
> > > > + BPF_EAPROG_SEC("cgroup_skb/egress/expected", BPF_PROG_TYPE_CGROUP_SKB,
> > > > + BPF_CGROUP_INET_EGRESS),
> > > > BPF_APROG_SEC("cgroup_skb/egress", BPF_PROG_TYPE_CGROUP_SKB,
> > > > BPF_CGROUP_INET_EGRESS),
> >
> > are you saying that when bpf prog has SEC("cgroup_skb/egress",.. libbpf actually
> > _not_ passing BPF_CGROUP_INET_EGRESS as expected_attach to the kernel?
>
> Yes, that seems to be the difference between BPF_EAPROG_SEC and BPF_APROG_SEC.
Yeah, "EA" version adds expected_attach_type ("E" stands for "expected")
at load time and "A" version only specifies attach type to use at attach
time (stands for "attach").
>
> > I think it's a libbpf bug and not something to workaround with retries.
>
> This predates me, but I assume it's a backwards-compatibility move.
> Because older kernels might know about expected_attach_type, but still
> allow ingress/egress programs to be attached. I'm fine with dropping
Only egress.
ingress doesn't have that problem.
> that (I actually had to work around this problem in
> bpf_program__attach_cgroup), but if anyone is feeling strongly about
> tiny chance of breaking something, we'll have to teach libbpf to retry
> load without expected_attach_type, if that one fails (which fails in
> its own way, so I'd rather not do it).
Answered backward compatibility point in the previous e-mail.
--
Andrey Ignatov
