On Wed, Jun 3, 2020 at 8:39 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > > maharishi bhargava <bhargavamaharishi@xxxxxxxxx> writes: > > > On Wed, Jun 3, 2020 at 4:41 PM Maciej Fijalkowski > > <maciej.fijalkowski@xxxxxxxxx> wrote: > >> > >> On Wed, Jun 03, 2020 at 01:07:05PM +0200, Toke Høiland-Jørgensen wrote: > >> > Maciej Fijalkowski <maciej.fijalkowski@xxxxxxxxx> writes: > >> > > >> > > On Wed, Jun 03, 2020 at 12:49:25PM +0200, Toke Høiland-Jørgensen wrote: > >> > >> maharishi bhargava <bhargavamaharishi@xxxxxxxxx> writes: > >> > >> > >> > >> > On Tue, Jun 2, 2020 at 9:31 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > >> > >> >> > >> > >> >> maharishi bhargava <bhargavamaharishi@xxxxxxxxx> writes: > >> > >> >> > >> > >> >> > On Tue 2 Jun, 2020, 14:31 Toke Høiland-Jørgensen, <toke@xxxxxxxxxx> wrote: > >> > >> >> >> > >> > >> >> >> maharishi bhargava <bhargavamaharishi@xxxxxxxxx> writes: > >> > >> >> >> > >> > >> >> >> > Hi, in my XDP program, I want to redirect some packets using AF_XDP > >> > >> >> >> > and redirect other packets directly from driver space. > >> > >> >> >> > Redirection through AF_XDP works fine, but redirection through dev map > >> > >> >> >> > stops after some packets are processed. > >> > >> >> >> > >> > >> >> >> Do you mean it stops even if you are *only* redirecting to a devmap, or > >> > >> >> >> if you are first redirecting a few packets to AF_XDP, then to devmap? > >> > >> >> >> > >> > >> >> >> Also, which driver(s) are the physical NICs you're redirecting to/from > >> > >> >> >> using, and which kernel version are you on? > >> > >> >> >> > >> > >> >> >> -Toke > >> > >> >> > > >> > >> >> > > >> > >> >> > > >> > >> >> > Currently, I'm trying to redirect packets only using devmap. But also > >> > >> >> > have code for redirection using AF_XDP(only when a given condition is > >> > >> >> > satisfied). A DPDK program is running in userspace which will receive > >> > >> >> > packets from AF_XDP. > >> > >> >> > >> > >> >> Right, so it's just devmap redirect that breaks. What do you mean > >> > >> >> 'redirection stops', exactly? How are you seeing this? Does xdp_monitor > >> > >> >> (from samples/bpf) report any exceptions? > >> > >> >> > >> > >> >> -Toke > >> > >> >> > >> > >> > So, In my setup, there are three systems, Let's Assume A, B, C. System > >> > >> > B is acting as a forwarder between A and C. So I can see the number of > >> > >> > packets received at system C. To be specific, only 1024 packets are > >> > >> > received. If I remove the xsks_map part from the code and don't run > >> > >> > DPDK in userspace. This problem does not occur. Also if I forward all > >> > >> > the packets using AF_XDP, there is no such issue. > >> > >> > >> > >> I thought you said you were seeing the problem when only redirecting to > >> > >> a devmap? So why does the xsk_map code impact this? I think you may have > >> > >> to share some code... > >> > > > >> > > Isn't the case here that either xsk_map or dev_map consumes the frame and > >> > > therefore the latter doesn't see it? so cloning might be needed here? > >> > > >> > Yeah, certainly you can't redirect *the same packet* to both xsk_map and > >> > devmap - but that wasn't what I understood was the use case here? > >> > >> Maybe the best would be if Maharishi shared the code as you requested :) > >> > >> > > >> > -Toke > >> > > > CODE: > > BPF MAPS: > > > > > > struct bpf_map_def SEC("maps") xsks_map = { > > .type = BPF_MAP_TYPE_XSKMAP, > > .key_size = sizeof(int), > > .value_size = sizeof(int), > > .max_entries = 64, /* Assume netdev has no more than 64 queues */ > > }; > > > > struct bpf_map_def SEC("maps") tx_port = { > > .type = BPF_MAP_TYPE_DEVMAP, > > .key_size = sizeof(int), > > .value_size = sizeof(int), > > .max_entries = 1024, > > }; > > > > struct Ingress_qos_lts_value{ > > struct bpf_spin_lock lock; > > u64 timestamp; > > }; > > struct bpf_map_def SEC("maps") Ingress_qos_lts = { > > .type = BPF_MAP_TYPE_ARRAY, > > .key_size = sizeof(u32), > > .value_size = sizeof(struct Ingress_qos_lts_value), > > .max_entries = 1025, > > }; > > BPF_ANNOTATE_KV_PAIR(Ingress_qos_lts,u32,struct Ingress_qos_lts_value); > > > > > > SEC("prog") > > int ebpf_filter(struct xdp_md *ctx){ > > struct xdp_output xout; > > xout.output_port = 1; > > void* ebpf_packetStart = ((void*)(long)ctx->data); > > void* ebpf_packetEnd = ((void*)(long)ctx->data_end); > > u64 rate = 100;//100 Kbps > > rate *= 1000*1000*100;//10 Gbps > > u32 key = 1;//some key > > u64 packet_length=(ebpf_packetEnd-ebpf_packetStart-42)*8; > > packet_length *= 1000000000; //packet length * 10^9, to convert > > rate from second to nanosecond > > struct Ingress_qos_lts_value* val; > > val = bpf_map_lookup_elem(&Ingress_qos_lts, &key); > > u64 now = bpf_ktime_get_ns(); > > u64 lts; > > if (val) { > > bpf_spin_lock(&val->lock); > > lts = *(&val->timestamp)+(packet_length/rate); > > if(now>lts){ > > lts = now; > > } > > *(&val->timestamp) = lts; > > bpf_spin_unlock(&val->lock); > > // printk("Time : %x %x\n",lts,now); > > if(lts>now){ > > return bpf_redirect_map(&xsks_map, ctx->rx_queue_index, 0); > > } > > } > > return bpf_redirect_map(&tx_port,xout.output_port,0); > > } > > > > So, Basically this code redirects the packet to some other interface > > or sends the packet to userspace based on the incoming packet rate. > > Well, if you say it goes away when you remove the xsk code, the obvious > explanation would be that the packets are being redirected to userspace > instead? What does xdp_monitor say? > > -Toke > No packets are not going to userspace. NIC stops processing any more packets after 1024 redirected packets. I'll post the results of xdp_monitor asap. Also, one piece of information that might be helpful. In DPDK's default code for creating xsk_socket, the value of bind_flags was 0. When I changed it to XDP_COPY(1 << 1), everything started working correctly. So, is it something like, the socket was getting created in zero-copy mode and as far as I know, kernel version 5.3 does not support zero-copy mode. Due to this xdp_redirect using DEVMAP was not working as expected. - Maharishi
![]() |