Here's an update -
On Fri, May 7, 2021 at 8:17 PM Srivats P <pstavirs@xxxxxxxxx> wrote:
>
> On Mon, May 3, 2021 at 1:54 PM Magnus Karlsson
> <magnus.karlsson@xxxxxxxxx> wrote:
> >
> > On Thu, Apr 29, 2021 at 5:47 PM Srivats P <pstavirs@xxxxxxxxx> wrote:
> > >
> > > On Tue, Apr 27, 2021 at 12:58 PM Magnus Karlsson
> > > <magnus.karlsson@xxxxxxxxx> wrote:
> > > >
> > > > On Fri, Apr 23, 2021 at 5:44 PM Srivats P <pstavirs@xxxxxxxxx> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm using sendto() to kick tx in my AF_XDP program after I submit
> > > > > descriptors to the tx ring -
> > > > >
> > > > > ret = sendto(xsk_socket__fd(xsk_), NULL, 0, MSG_DONTWAIT, NULL, 0);
> > > > >
> > > > > However, I'm receiving EPERM as the return value every time. AFAIK
> > > > > this is not an expected return value. Since this is with i40e, I
> > > > > checked i40e_xsk_wakeup() - but that also doesn't return EPERM. I am
> > > > > running as root and I don't see any problems with creating the xsk,
> > > > > configuring umem etc.
> > > > >
> > > > > Also, no packets seem to go out either.
> > > > >
> > > > > # uname -a
> > > > > Linux Ostinato-1 5.11.15-1-default #1 SMP Fri Apr 16 16:47:34 UTC 2021
> > > > > (64fb5bf) x86_64 x86_64 x86_64 GNU/Linux
> > > > >
> > > > > I don't see the problem on another machine with i40e but older kernel 5.4 series
> > > > >
> > > > > Any suggestions on what to look for or how to proceed?
> > > >
> > > > Weird. Have not seen this before. What is your command line for
> > > > xdpsock? Is it unmodified?
> > >
> > > This is not xdpsock, but my own AF_XDP program.
> > >
> > > >
> > > > Using bpftrace, we can get the call stack of xsk_sendmsg. Somewhere in
> > > > this stack there must be an EPERM. You can run the same command on
> > > > your system, but use ftrace to see what a sendto call hits. Then see
> > > > where the code terminates.
> > > >
> > > > mkarlsso@kurt:~/src/dna-linux$ sudo bpftrace -e 'kprobe:xsk_sendmsg {
> > > > @[kstack()] = count(); }'
> > > > Attaching 1 probe...
> > > > ^C
> > > >
> > > > @[
> > > > xsk_sendmsg+1
> > > > sock_sendmsg+94
> > > > __sys_sendto+238
> > > > __x64_sys_sendto+37
> > > > do_syscall_64+51
> > > > entry_SYSCALL_64_after_hwframe+68
> > > > ]: 2244805
> > >
> > > Ostinato-1:~ # bpftrace -e 'kprobe:xsk_sendmsg {
> > > @[kstack()] = count(); }'
> > > Attaching 1 probe...^C@[
> > > xsk_sendmsg+1
> > > sock_sendmsg+94
> > > __sys_sendto+238
> > > __x64_sys_sendto+37
> > > do_syscall_64+51
> > > entry_SYSCALL_64_after_hwframe+68
> > > ]: 1253307
> > >
> > > Which doesn't seem to suggest any error - I've looked at the source
> > > code for all these functions, but don't see any reference to EPERM.
> >
> > It must be in there somewhere :-). Could you plesae use ftrace
> > (through perf for example) and trace all functions that a sendto hits
> > in your case? Then we might see what it hits.
> >
> > Are you running in SKB mode or in zero-copy mode? Guess it is
> > zero-copy from your mail, but just want to verify. Does Rx work as
> > expected?
> >
> > Could you share your AF_XDP program?
>
> After some experimentation and a lot of head-scratching, I found part
> of the problem last night. The sendto() was not returning EPERM (-1),
> but ENXIO (-6) - I was mistakenly printing the return value of the
> sento() call (which always returns -1 in case of failure), instead of
> errno (duh!).
>
> Looking at the code, I see ENXIO is returned if the xsk is unbound.
> I'm still investigating this and will post an update soon. The problem
> is happening at a customer end and there's some delay and follow up
> required to get the logs.
sendto() was returning ENXIO because the interface MTU was set to 9000
which I know is not supported with AF_XDP. But shouldn't
xsk_socket__create() fail in this case? Note the actual packet being
transmitted was 64 bytes.
Not sure if it has a role in the above sendto() failure, but before
xsk socket create, my call to bpf_set_link_xdp_fd() was failing
because of the MTU problem (the newly added error message for this
case was very helpful!). Once MTU was reduced to 1500 both the RX eBPF
program link to the interface failure and the TX sendto() returning
ENXIO always went away. Kernel version 5.12
Can someone tell me what is expected to happen for a Tx AF_XDP socket
in case of MTU > 4K?
I also found a second case of sendto() returning ENXIO. In this
scenario, I was removing my RX eBPF program by calling
bpf_set_link_xdp_fd(ifIndex, -1, 0)
while AF_XDP transmit (and associated sento() wakeup) was still going
on. In this case, sendto starts failing with ENETDOWN for some time
followed by ENXIO subsequently. This case was on Kernel version 5.4.0
Does removing a XDP program cause the interface to go down (ENETDOWN)
leading to XDP socket unbind (ENXIO)? Should removing (or replacing)
an RX eBPF program, affect AF_XDP TX?
Srivats
|