Hi Eric, Thanks for a detailed reply. Sorry I couldn't reply as I was completely bed ridden. In order for me to try this, I require few inputs (as I am new to all this)... 1. How do I register in Kernel, that my eBPF program should be called? Is this https://netdevconf.info/1.1/proceedings/papers/On-getting-tc-classifier-fully-programmable-with-cls-bpf.pdf and http://man7.org/linux/man-pages/man8/tc-bpf.8.html correct documents ? 2. Some info with respect to EDT and skb->tstamp and how things work. On Mon, Aug 26, 2019 at 12:02 PM Eric Dumazet <eric.dumazet@xxxxxxxxx> wrote: > > > > On 8/25/19 7:52 PM, Cong Wang wrote: > > On Wed, Aug 21, 2019 at 11:00 PM Akshat Kakkar <akshat.1984@xxxxxxxxx> wrote: > >> > >> On Thu, Aug 22, 2019 at 3:37 AM Cong Wang <xiyou.wangcong@xxxxxxxxx> wrote: > >>>> I am using ipset + iptables to classify and not filters. Besides, if > >>>> tc is allowing me to define qdisc -> classes -> qdsic -> classes > >>>> (1,2,3 ...) sort of structure (ie like the one shown in ascii tree) > >>>> then how can those lowest child classes be actually used or consumed? > >>> > >>> Just install tc filters on the lower level too. > >> > >> If I understand correctly, you are saying, > >> instead of : > >> tc filter add dev eno2 parent 100: protocol ip prio 1 handle > >> 0x00000001 fw flowid 1:10 > >> tc filter add dev eno2 parent 100: protocol ip prio 1 handle > >> 0x00000002 fw flowid 1:20 > >> tc filter add dev eno2 parent 100: protocol ip prio 1 handle > >> 0x00000003 fw flowid 2:10 > >> tc filter add dev eno2 parent 100: protocol ip prio 1 handle > >> 0x00000004 fw flowid 2:20 > >> > >> > >> I should do this: (i.e. changing parent to just immediate qdisc) > >> tc filter add dev eno2 parent 1: protocol ip prio 1 handle 0x00000001 > >> fw flowid 1:10 > >> tc filter add dev eno2 parent 1: protocol ip prio 1 handle 0x00000002 > >> fw flowid 1:20 > >> tc filter add dev eno2 parent 2: protocol ip prio 1 handle 0x00000003 > >> fw flowid 2:10 > >> tc filter add dev eno2 parent 2: protocol ip prio 1 handle 0x00000004 > >> fw flowid 2:20 > > > > > > Yes, this is what I meant. > > > > > >> > >> I tried this previously. But there is not change in the result. > >> Behaviour is exactly same, i.e. I am still getting 100Mbps and not > >> 100kbps or 300kbps > >> > >> Besides, as I mentioned previously I am using ipset + skbprio and not > >> filters stuff. Filters I used just to test. > >> > >> ipset -N foo hash:ip,mark skbinfo > >> > >> ipset -A foo 10.10.10.10, 0x0x00000001 skbprio 1:10 > >> ipset -A foo 10.10.10.20, 0x0x00000002 skbprio 1:20 > >> ipset -A foo 10.10.10.30, 0x0x00000003 skbprio 2:10 > >> ipset -A foo 10.10.10.40, 0x0x00000004 skbprio 2:20 > >> > >> iptables -A POSTROUTING -j SET --map-set foo dst,dst --map-prio > > > > Hmm.. > > > > I am not familiar with ipset, but it seems to save the skbprio into > > skb->priority, so it doesn't need TC filter to classify it again. > > > > I guess your packets might go to the direct queue of HTB, which > > bypasses the token bucket. Can you dump the stats and check? > > With more than 64K 'classes' I suggest to use a single FQ qdisc [1], and > an eBPF program using EDT model (Earliest Departure Time) > > The BPF program would perform the classification, then find a data structure > based on the 'class', and then update/maintain class virtual times and skb->tstamp > > TBF = bpf_map_lookup_elem(&map, &classid); > > uint64_t now = bpf_ktime_get_ns(); > uint64_t time_to_send = max(TBF->time_to_send, now); > > time_to_send += (u64)qdisc_pkt_len(skb) * NSEC_PER_SEC / TBF->rate; > if (time_to_send > TBF->max_horizon) { > return TC_ACT_SHOT; > } > TBF->time_to_send = time_to_send; > skb->tstamp = max(time_to_send, skb->tstamp); > if (time_to_send - now > TBF->ecn_horizon) > bpf_skb_ecn_set_ce(skb); > return TC_ACT_OK; > > tools/testing/selftests/bpf/progs/test_tc_edt.c shows something similar. > > > [1] MQ + FQ if the device is multi-queues. > > Note that this setup scales very well on SMP, since we no longer are forced > to use a single HTB hierarchy (protected by a single spinlock) >
