On Mon, 4 Dec 2023 at 16:42, Joshua Coutinho <souichirosano@xxxxxxxxx> wrote:
>
> Hi All,
>
> I'm trying to transmit udp packets via an xsk socket. (XDP receives
> work just fine). The xdp program is irrelevant/unused, I'm trying to
> simply leverage the xsk socket write.
>
> (Kernel: Linux fedora 6.5.12-300.fc39.x86_64)
> (OS: "Fedora Linux 39 (Workstation Edition)"
>
> I want a minimal working example of sending packets via an XSK socket
> over loopback in user space land. I want to be able to fill in the
> required memory regions and trigger the kernel to send the packet and
> capture the sent packets on the other side with 'nc -lu 127.0.0.1
> <port>' This seems to happen partly successfully but on the ingress
> part of the loopback it is dropped somewhere after reaching the kernel
> function ip_rcv and then nf_hook_slow. Specifically, I simply want to
> write a packet into a UMEM region, fill in the TX descriptor and then
> submit that descriptor like so.
>
> u32 txIdx = -1;
> const u32 txSlotsRecvd = xsk_ring_prod__reserve(&qs.txQ, 1, &txIdx);
> u32 addr = umem.txState.nextSlot();
>
> xdp_desc* txDescr = xsk_ring_prod__tx_desc(&qs.txQ, txIdx);
> txDescr->addr = addr;
> txDescr->len = sizeof(OrderFrame);
> txDescr->options = 0;
>
> u8* outputBuf = umem.buffer + addr;
>
> TimeNs submitTime = currentTimeNs();
> OrderFrame& frame = *reinterpret_cast<OrderFrame *>(outputBuf);
>
> std::array<u8, ETH_ALEN> sourceMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
> std::array<u8, ETH_ALEN> destMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
> std::copy(sourceMac.begin(), sourceMac.end(), frame.eth.h_source);
> std::copy(destMac.begin(), destMac.end(), frame.eth.h_dest);
>
> frame.eth.h_proto = htons(ETH_P_IP);
> frame.ip.ihl = 5;
> frame.ip.version = 4;
> frame.ip.tos = 0;
> frame.ip.tot_len = htons(sizeof(OrderFrame) - sizeof(ethhdr));
> frame.ip.id = orderId;
>
> frame.ip.frag_off = 0x0;
> frame.ip.ttl = static_cast<u8>(255);
> frame.ip.protocol = 17;
> frame.ip.check = 0;
> constexpr u8 sourceIPBytes[4] = {127, 0, 0, 1};
> constexpr u8 destIPBytes[4] = {127, 0, 0, 1};
> const u32 sourceIP = *reinterpret_cast<const u32*>(sourceIPBytes);
> const u32 destIP = *reinterpret_cast<const u32*>(destIPBytes);
> frame.ip.saddr = sourceIP;
> frame.ip.daddr = destIP;
> const u8* dataptr = reinterpret_cast<u8 *>(&frame.ip);
> const u16 kernelcsum = ip_fast_csum(dataptr, frame.ip.ihl);
> frame.ip.check = kernelcsum;
> constexpr int udpPacketSz = sizeof(OrderFrame) -
> sizeof(ethhdr) - sizeof(iphdr);
> frame.udp.len = htons(udpPacketSz);
> frame.udp.check = 0;
> frame.udp.dest = htons(OE_PORT);
> frame.udp.source = htons(1234);
> ... // application packet logic
> frame.udp.check = 0;
>
> xsk_ring_prod__submit(&qs.txQ, 1);
> if (xsk_ring_prod__needs_wakeup(&qs.txQ)) {
> assert((socket.cfg.bind_flags & XDP_COPY) != 0);
> const ssize_t ret = sendto(socket.xskFD, nullptr, 0,
> MSG_DONTWAIT, nullptr, 0);
> }
>
> This is a relevant stacktrace from the kernel indicating the path of
> my packet after the above sendto is called.
>
> __netif_receive_skb_one_core+0x3c/0xa0
> process_backlog+0x85/0x120
> __napi_poll+0x28/0x1b0
> net_rx_action+0x2a4/0x380
> __do_softirq+0xd1/0x2c8
> do_softirq.part.0+0x3d/0x60
> __local_bh_enable_ip+0x68/0x70
> __dev_direct_xmit+0x152/0x210
> __xsk_generic_xmit+0x3e4/0x710
> xsk_sendmsg+0x12f/0x1f0
> __sys_sendto+0x1d6/0x1e0
> __x64_sys_sendto+0x24/0x30
> do_syscall_64+0x5d/0x90
> entry_SYSCALL_64_after_hwframe+0x6e/0xd8
>
> My socket is bound to localhost using xdpgeneric. I see the
> transmitted packets in tcpdump and via bpftrace I see that ip_rcv is
> invoked for the packets. nf_hook_slow is also invoked with 1 active
> prerouting hook. On kfree_skb I see the reason for the drop is 'reason
> not specified'. Examining the packet in tcpdump I see no errors with
> the checksums or packet lengths and ports. Listeners for the
> corresponding udp ports never receieve the packets.This is how I
> create my socket
>
> cfg.rx_size = XSKQueues::NUM_READ_DESC;
> cfg.tx_size = XSKQueues::NUM_WRITE_DESC;
> cfg.libxdp_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD;
> cfg.xdp_flags = XDP_FLAGS_SKB_MODE;
> cfg.bind_flags = XDP_USE_NEED_WAKEUP | XDP_COPY;
>
> if (xsk_socket__create(&socket, iface.c_str(), QUEUE, umem.umem,
> &qs.rxQ, &qs.txQ, &cfg)) {
> perror("XSK: ");
> exit(EXIT_FAILURE);
> }
>
> What could be the issue?
It is the right forum. Sorry, just forgot to answer your mail.
What happens if you instead of opening an AF_INET socket, open an
AF_PACKET socket (or an AF_XDP socket for that matter) at the
receiving end? Do you get the packet? If so, might it just be that
your packet is wrong and it is being discarded by the stack? Use
tcpdump or wireshark at the receiving end and see what it says. That
is if you get a packet out of the AF_PACKET socket.
|