Hi All, I'm trying to transmit udp packets via an xsk socket. (XDP receives work just fine). The xdp program is irrelevant/unused, I'm trying to simply leverage the xsk socket write. (Kernel: Linux fedora 6.5.12-300.fc39.x86_64) (OS: "Fedora Linux 39 (Workstation Edition)" I want a minimal working example of sending packets via an XSK socket over loopback in user space land. I want to be able to fill in the required memory regions and trigger the kernel to send the packet and capture the sent packets on the other side with 'nc -lu 127.0.0.1 <port>' This seems to happen partly successfully but on the ingress part of the loopback it is dropped somewhere after reaching the kernel function ip_rcv and then nf_hook_slow. Specifically, I simply want to write a packet into a UMEM region, fill in the TX descriptor and then submit that descriptor like so. u32 txIdx = -1; const u32 txSlotsRecvd = xsk_ring_prod__reserve(&qs.txQ, 1, &txIdx); u32 addr = umem.txState.nextSlot(); xdp_desc* txDescr = xsk_ring_prod__tx_desc(&qs.txQ, txIdx); txDescr->addr = addr; txDescr->len = sizeof(OrderFrame); txDescr->options = 0; u8* outputBuf = umem.buffer + addr; TimeNs submitTime = currentTimeNs(); OrderFrame& frame = *reinterpret_cast<OrderFrame *>(outputBuf); std::array<u8, ETH_ALEN> sourceMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}; std::array<u8, ETH_ALEN> destMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}; std::copy(sourceMac.begin(), sourceMac.end(), frame.eth.h_source); std::copy(destMac.begin(), destMac.end(), frame.eth.h_dest); frame.eth.h_proto = htons(ETH_P_IP); frame.ip.ihl = 5; frame.ip.version = 4; frame.ip.tos = 0; frame.ip.tot_len = htons(sizeof(OrderFrame) - sizeof(ethhdr)); frame.ip.id = orderId; frame.ip.frag_off = 0x0; frame.ip.ttl = static_cast<u8>(255); frame.ip.protocol = 17; frame.ip.check = 0; constexpr u8 sourceIPBytes[4] = {127, 0, 0, 1}; constexpr u8 destIPBytes[4] = {127, 0, 0, 1}; const u32 sourceIP = *reinterpret_cast<const u32*>(sourceIPBytes); const u32 destIP = *reinterpret_cast<const u32*>(destIPBytes); frame.ip.saddr = sourceIP; frame.ip.daddr = destIP; const u8* dataptr = reinterpret_cast<u8 *>(&frame.ip); const u16 kernelcsum = ip_fast_csum(dataptr, frame.ip.ihl); frame.ip.check = kernelcsum; constexpr int udpPacketSz = sizeof(OrderFrame) - sizeof(ethhdr) - sizeof(iphdr); frame.udp.len = htons(udpPacketSz); frame.udp.check = 0; frame.udp.dest = htons(OE_PORT); frame.udp.source = htons(1234); ... // application packet logic frame.udp.check = 0; xsk_ring_prod__submit(&qs.txQ, 1); if (xsk_ring_prod__needs_wakeup(&qs.txQ)) { assert((socket.cfg.bind_flags & XDP_COPY) != 0); const ssize_t ret = sendto(socket.xskFD, nullptr, 0, MSG_DONTWAIT, nullptr, 0); } This is a relevant stacktrace from the kernel indicating the path of my packet after the above sendto is called. __netif_receive_skb_one_core+0x3c/0xa0 process_backlog+0x85/0x120 __napi_poll+0x28/0x1b0 net_rx_action+0x2a4/0x380 __do_softirq+0xd1/0x2c8 do_softirq.part.0+0x3d/0x60 __local_bh_enable_ip+0x68/0x70 __dev_direct_xmit+0x152/0x210 __xsk_generic_xmit+0x3e4/0x710 xsk_sendmsg+0x12f/0x1f0 __sys_sendto+0x1d6/0x1e0 __x64_sys_sendto+0x24/0x30 do_syscall_64+0x5d/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 My socket is bound to localhost using xdpgeneric. I see the transmitted packets in tcpdump and via bpftrace I see that ip_rcv is invoked for the packets. nf_hook_slow is also invoked with 1 active prerouting hook. On kfree_skb I see the reason for the drop is 'reason not specified'. Examining the packet in tcpdump I see no errors with the checksums or packet lengths and ports. Listeners for the corresponding udp ports never receieve the packets.This is how I create my socket cfg.rx_size = XSKQueues::NUM_READ_DESC; cfg.tx_size = XSKQueues::NUM_WRITE_DESC; cfg.libxdp_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD; cfg.xdp_flags = XDP_FLAGS_SKB_MODE; cfg.bind_flags = XDP_USE_NEED_WAKEUP | XDP_COPY; if (xsk_socket__create(&socket, iface.c_str(), QUEUE, umem.umem, &qs.rxQ, &qs.txQ, &cfg)) { perror("XSK: "); exit(EXIT_FAILURE); } What could be the issue?
![]() |