On Sat, Feb 8, 2025 at 10:11 AM Martin KaFai Lau <martin.lau@xxxxxxxxx> wrote:
>
> On 2/7/25 4:07 AM, Jason Xing wrote:
> > On Fri, Feb 7, 2025 at 10:18 AM Jason Xing <kerneljasonxing@xxxxxxxxx> wrote:
> >>
> >> On Fri, Feb 7, 2025 at 10:07 AM Martin KaFai Lau <martin.lau@xxxxxxxxx> wrote:
> >>>
> >>> On 2/5/25 10:56 PM, Jason Xing wrote:
> >>>>>> I have to rephrase a bit in case Martin visits here soon: I will
> >>>>>> compare two approaches 1) reply value, 2) bpf kfunc and then see which
> >>>>>> way is better.
> >>>>>
> >>>>> I have already explained in details why the 1) reply value from the bpf prog
> >>>>> won't work. Please go back to that reply which has the context.
> >>>>
> >>>> Yes, of course I saw this, but I said I need to implement and dig more
> >>>> into this on my own. One of my replies includes a little code snippet
> >>>> regarding reply value approach. I didn't expect you to misunderstand
> >>>> that I would choose reply value, so I rephrase it like above :)
> >>>
> >>> I did see the code snippet which is incomplete, so I have to guess. afaik, it is
> >>> not going to work. I was hoping to save some time without detouring to the
> >>> reply-value path in case my earlier message was missed. I will stay quiet and
> >>> wait for v9 first then to avoid extending this long thread further.
> >>
> >> I see. I'm grateful that you point out the right path. I'm still
> >> investigating to find a good existing example in selftests and how to
> >> support kfunc.
> >
> > Martin, sorry to revive this thread.
> >
> > It's a little bit hard for me to find a proper example to follow. I
> > tried to call __bpf_kfunc in the BPF_SOCK_OPS_TS_SND_CB callback and
> > then failed because kfunc is not supported in the sock_ops case.
> > Later, I tried to kprobe to hook a function, say,
> > tcp_tx_timestamp_bpf(), passed the skb parameter to the kfunc and then
> > got an error.
> >
> > Here is code snippet:
> > 1) net/ipv4/tcp.c
> > +__bpf_kfunc static void tcp_init_tx_timestamp(struct sk_buff *skb)
> > +{
> > + struct skb_shared_info *shinfo = skb_shinfo(skb);
> > + struct tcp_skb_cb *tcb = TCP_SKB_CB(skb);
> > +
> > + printk(KERN_ERR "jason: %d, %d\n\n", tcb->txstamp_ack,
> > shinfo->tx_flags);
> > + /*
> > + tcb->txstamp_ack = 2;
> > + shinfo->tx_flags |= SKBTX_BPF;
> > + shinfo->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1;
> > + */
> > +}
> > Note: I skipped copying some codes like BTF_ID_FLAGS...
>
> This part is missing, so I can only guess again. This BTF_ID_FLAGS
> and the kfunc registration part went wrong when trying to add the
> new kfunc for the sock_ops program. There are kfunc examples for
> netdev related bpf prog in filter.c. e.g. bpf_sock_addr_set_sun_path.
>
> [ The same goes for another later message where the changes in
> bpf_skops_tx_timestamping is missing, so I won't comment there. ]
>
> >
> > 2) bpf prog
> > SEC("kprobe/tcp_tx_timestamp_bpf") // I wrote a new function/wrapper to hook
> > int BPF_KPROBE(kprobe__tcp_tx_timestamp_bpf, struct sock *sk, struct
> > sk_buff *skb)
> > {
> > tcp_init_tx_timestamp(skb);
> > return 0;
> > }
> >
> > Then running the bpf prog, I got the following message:
> > ; tcp_init_tx_timestamp(skb); @ so_timestamping.c:281
> > 1: (85) call tcp_init_tx_timestamp#120682
> > arg#0 pointer type STRUCT sk_buff must point to scalar, or struct with scalar
> > processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > peak_states 0 mark_read 0
> > -- END PROG LOAD LOG --
> > libbpf: prog 'kprobe__tcp_tx_timestamp_bpf': failed to load: -22
> > libbpf: failed to load object 'so_timestamping'
> > libbpf: failed to load BPF skeleton 'so_timestamping': -22
> > test_so_timestamping:FAIL:open and load skel unexpected error: -22
> >
> > If I don't pass any parameter in the kfunc, it can work.
> >
> > Should we support the sock_ops for __bpf_kfunc?
>
> sock_ops does support kfunc. The patch 12 selftest is using the
> bpf_cast_to_kern_ctx() and it is a kfunc:
>
> --------8<--------
> BTF_KFUNCS_START(common_btf_ids)
> BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
> -------->8--------
>
> It just the new kfunc is not registered at the right place, so the verifier
> cannot find it.
>
> Untested code on top of your v8, so I don't have your latest
> changes on the txstamp_ack_bpf bits...etc.
Thanks for sharing your great understanding of BPF. And it's working!
Many thanks here.
>
> diff --git i/kernel/bpf/btf.c w/kernel/bpf/btf.c
> index 9433b6467bbe..740210f883dc 100644
> --- i/kernel/bpf/btf.c
> +++ w/kernel/bpf/btf.c
> @@ -8522,6 +8522,7 @@ static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type)
> case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
> case BPF_PROG_TYPE_CGROUP_SOCKOPT:
> case BPF_PROG_TYPE_CGROUP_SYSCTL:
> + case BPF_PROG_TYPE_SOCK_OPS:
The above line is exactly what I want (before this, I had no clue
about how to write this part), causing my whole kfunc feature not to
work.
> return BTF_KFUNC_HOOK_CGROUP;
> case BPF_PROG_TYPE_SCHED_ACT:
> return BTF_KFUNC_HOOK_SCHED_ACT;
> diff --git i/net/core/filter.c w/net/core/filter.c
> index d3395ffe058e..3bad67eb5c9e 100644
> --- i/net/core/filter.c
> +++ w/net/core/filter.c
> @@ -12102,6 +12102,30 @@ __bpf_kfunc int bpf_sk_assign_tcp_reqsk(struct __sk_buff *s, struct sock *sk,
> #endif
> }
>
> +enum {
> + BPF_SOCK_OPS_TX_TSTAMP_TCP_ACK = 1 << 0,
> +};
Could I remove this flag since we have BPF_SOCK_OPS_TS_ACK_OPT_CB to
control whether to report or not?
> +
> +__bpf_kfunc int bpf_sock_ops_enable_tx_tstamp(struct bpf_sock_ops_kern *skops, int flags)
> +{
> + struct sk_buff *skb;
> +
> + if (skops->op != BPF_SOCK_OPS_TS_SND_CB)
> + return -EOPNOTSUPP;
> +
> + if (flags & ~BPF_SOCK_OPS_TX_TSTAMP_TCP_ACK)
> + return -EINVAL;
> +
> + skb = skops->skb;
> + /* [REMOVE THIS COMMENT]: sk_is_tcp check will be needed in the future */
> + if (flags & BPF_SOCK_OPS_TX_TSTAMP_TCP_ACK)
> + TCP_SKB_CB(skb)->txstamp_ack_bpf = 1;
> + skb_shinfo(skb)->tx_flags |= SKBTX_BPF;
> + skb_shinfo(skb)->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1;
> +
> + return 0;
> +}
> +
> __bpf_kfunc_end_defs();
>
> int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
> @@ -12135,6 +12159,10 @@ BTF_KFUNCS_START(bpf_kfunc_check_set_tcp_reqsk)
> BTF_ID_FLAGS(func, bpf_sk_assign_tcp_reqsk, KF_TRUSTED_ARGS)
> BTF_KFUNCS_END(bpf_kfunc_check_set_tcp_reqsk)
>
> +BTF_KFUNCS_START(bpf_kfunc_check_set_sock_ops)
> +BTF_ID_FLAGS(func, bpf_sock_ops_enable_tx_tstamp, KF_TRUSTED_ARGS)
> +BTF_KFUNCS_END(bpf_kfunc_check_set_sock_ops)
> +
> static const struct btf_kfunc_id_set bpf_kfunc_set_skb = {
> .owner = THIS_MODULE,
> .set = &bpf_kfunc_check_set_skb,
> @@ -12155,6 +12183,11 @@ static const struct btf_kfunc_id_set bpf_kfunc_set_tcp_reqsk = {
> .set = &bpf_kfunc_check_set_tcp_reqsk,
> };
>
> +static const struct btf_kfunc_id_set bpf_kfunc_set_sock_ops = {
> + .owner = THIS_MODULE,
> + .set = &bpf_kfunc_check_set_sock_ops,
> +};
> +
> static int __init bpf_kfunc_init(void)
> {
> int ret;
> @@ -12173,6 +12206,7 @@ static int __init bpf_kfunc_init(void)
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &bpf_kfunc_set_xdp);
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SOCK_ADDR,
> &bpf_kfunc_set_sock_addr);
> + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SOCK_OPS, &bpf_kfunc_set_sock_ops);
> return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_kfunc_set_tcp_reqsk);
> }
