On Wed, Mar 6, 2024 at 11:39 AM Andrii Nakryiko <andrii@xxxxxxxxxx> wrote:
>
> prog->aux->sleepable is checked very frequently as part of (some) BPF
> program run hot paths. So this extra aux indirection seems wasteful and
> on busy systems might cause unnecessary memory cache misses.
>
> Let's move sleepable flag into prog itself to eliminate unnecessary
> pointer dereference.
>
> Signed-off-by: Andrii Nakryiko <andrii@xxxxxxxxxx>
> ---
> include/linux/bpf.h | 8 ++++----
> kernel/bpf/bpf_iter.c | 4 ++--
> kernel/bpf/core.c | 2 +-
> kernel/bpf/syscall.c | 6 +++---
> kernel/bpf/trampoline.c | 4 ++--
> kernel/bpf/verifier.c | 12 ++++++------
> kernel/events/core.c | 2 +-
> kernel/trace/bpf_trace.c | 2 +-
> net/bpf/bpf_dummy_struct_ops.c | 2 +-
> 9 files changed, 21 insertions(+), 21 deletions(-)
>
[...]
> @@ -2196,7 +2196,7 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred)
> btf_put(prog->aux->attach_btf);
>
> if (deferred) {
> - if (prog->aux->sleepable)
> + if (prog->sleepable)
> call_rcu_tasks_trace(&prog->aux->rcu, __bpf_prog_put_rcu);
> else
> call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
> @@ -2761,11 +2761,11 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
> }
>
> prog->expected_attach_type = attr->expected_attach_type;
> + prog->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
there is a huge difference between these two lines, got to be more
careful with bit fields here:
- prog->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
+ prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
> prog->aux->attach_btf = attach_btf;
> prog->aux->attach_btf_id = attr->attach_btf_id;
> prog->aux->dst_prog = dst_prog;
> prog->aux->dev_bound = !!attr->prog_ifindex;
> - prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
> prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
>
> /* move token into prog->aux, reuse taken refcnt */
[...]
