Hi All,
I'm trying to transmit udp packets via an xsk socket. (XDP receives
work just fine). The xdp program is irrelevant/unused, I'm trying to
simply leverage the xsk socket write.
(Kernel: Linux fedora 6.5.12-300.fc39.x86_64)
(OS: "Fedora Linux 39 (Workstation Edition)"
I want a minimal working example of sending packets via an XSK socket
over loopback in user space land. I want to be able to fill in the
required memory regions and trigger the kernel to send the packet and
capture the sent packets on the other side with 'nc -lu 127.0.0.1
<port>' This seems to happen partly successfully but on the ingress
part of the loopback it is dropped somewhere after reaching the kernel
function ip_rcv and then nf_hook_slow. Specifically, I simply want to
write a packet into a UMEM region, fill in the TX descriptor and then
submit that descriptor like so.
u32 txIdx = -1;
const u32 txSlotsRecvd = xsk_ring_prod__reserve(&qs.txQ, 1, &txIdx);
u32 addr = umem.txState.nextSlot();
xdp_desc* txDescr = xsk_ring_prod__tx_desc(&qs.txQ, txIdx);
txDescr->addr = addr;
txDescr->len = sizeof(OrderFrame);
txDescr->options = 0;
u8* outputBuf = umem.buffer + addr;
TimeNs submitTime = currentTimeNs();
OrderFrame& frame = *reinterpret_cast<OrderFrame *>(outputBuf);
std::array<u8, ETH_ALEN> sourceMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
std::array<u8, ETH_ALEN> destMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
std::copy(sourceMac.begin(), sourceMac.end(), frame.eth.h_source);
std::copy(destMac.begin(), destMac.end(), frame.eth.h_dest);
frame.eth.h_proto = htons(ETH_P_IP);
frame.ip.ihl = 5;
frame.ip.version = 4;
frame.ip.tos = 0;
frame.ip.tot_len = htons(sizeof(OrderFrame) - sizeof(ethhdr));
frame.ip.id = orderId;
frame.ip.frag_off = 0x0;
frame.ip.ttl = static_cast<u8>(255);
frame.ip.protocol = 17;
frame.ip.check = 0;
constexpr u8 sourceIPBytes[4] = {127, 0, 0, 1};
constexpr u8 destIPBytes[4] = {127, 0, 0, 1};
const u32 sourceIP = *reinterpret_cast<const u32*>(sourceIPBytes);
const u32 destIP = *reinterpret_cast<const u32*>(destIPBytes);
frame.ip.saddr = sourceIP;
frame.ip.daddr = destIP;
const u8* dataptr = reinterpret_cast<u8 *>(&frame.ip);
const u16 kernelcsum = ip_fast_csum(dataptr, frame.ip.ihl);
frame.ip.check = kernelcsum;
constexpr int udpPacketSz = sizeof(OrderFrame) -
sizeof(ethhdr) - sizeof(iphdr);
frame.udp.len = htons(udpPacketSz);
frame.udp.check = 0;
frame.udp.dest = htons(OE_PORT);
frame.udp.source = htons(1234);
... // application packet logic
frame.udp.check = 0;
xsk_ring_prod__submit(&qs.txQ, 1);
if (xsk_ring_prod__needs_wakeup(&qs.txQ)) {
assert((socket.cfg.bind_flags & XDP_COPY) != 0);
const ssize_t ret = sendto(socket.xskFD, nullptr, 0,
MSG_DONTWAIT, nullptr, 0);
}
This is a relevant stacktrace from the kernel indicating the path of
my packet after the above sendto is called.
__netif_receive_skb_one_core+0x3c/0xa0
process_backlog+0x85/0x120
__napi_poll+0x28/0x1b0
net_rx_action+0x2a4/0x380
__do_softirq+0xd1/0x2c8
do_softirq.part.0+0x3d/0x60
__local_bh_enable_ip+0x68/0x70
__dev_direct_xmit+0x152/0x210
__xsk_generic_xmit+0x3e4/0x710
xsk_sendmsg+0x12f/0x1f0
__sys_sendto+0x1d6/0x1e0
__x64_sys_sendto+0x24/0x30
do_syscall_64+0x5d/0x90
entry_SYSCALL_64_after_hwframe+0x6e/0xd8
My socket is bound to localhost using xdpgeneric. I see the
transmitted packets in tcpdump and via bpftrace I see that ip_rcv is
invoked for the packets. nf_hook_slow is also invoked with 1 active
prerouting hook. On kfree_skb I see the reason for the drop is 'reason
not specified'. Examining the packet in tcpdump I see no errors with
the checksums or packet lengths and ports. Listeners for the
corresponding udp ports never receieve the packets.This is how I
create my socket
cfg.rx_size = XSKQueues::NUM_READ_DESC;
cfg.tx_size = XSKQueues::NUM_WRITE_DESC;
cfg.libxdp_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD;
cfg.xdp_flags = XDP_FLAGS_SKB_MODE;
cfg.bind_flags = XDP_USE_NEED_WAKEUP | XDP_COPY;
if (xsk_socket__create(&socket, iface.c_str(), QUEUE, umem.umem,
&qs.rxQ, &qs.txQ, &cfg)) {
perror("XSK: ");
exit(EXIT_FAILURE);
}
What could be the issue?
|