Hi, this question has been discussed a bit on a different mailing list[1], but the information given there[2][3] has not yet reached the LARTC list. So here are URLs that may or may not provide insight into the problem, but no solution so far: [1] https://lists.bufferbloat.net/listinfo/bloat [2] https://lists.bufferbloat.net/pipermail/bloat/2021-June/016440.html [3] https://lists.bufferbloat.net/pipermail/bloat/2021-June/016441.html I myself have no idea regarding the problem, sorry. Thanks, Erik -- In the beginning, there was static routing. -- RFC 1118 On Wed, Jun 16, 2021 at 12:43:12PM +0100, cronolog+lartc wrote: > Hi, > > The initial query mentioned the issue occurs at around 4.5Gbps throughput or higher. > > I wonder if a 32-bit value is overflowing somewhere. If the traffic rate or some other > value used for internal calculations of the qdisc uses an unsigned 32-bit value, then > this can only go up to 4294967295 which is just short of 4.3Gbps. So would make sense > problems start to occur when you go beyond this throughput rate. I don't think the > filters or quantity of filters would be the problem, they just classify the traffic. > The problem would likely be within how a qdisc is implemented and how and when it > chooses to dequeue packets for transmission. More likely the bug is within HTB than > SFQ, as I don't think SFQ cares about throughput rate, but that's just my guess. > > How are the qdiscs/classes created? It would be interesting to compare with the > configuration from the person who has it working at 5Gbps. > > I also thought tc/iproute2 supported 64-bit values for traffic rate. The tc man page > says otherwise i.e. only 32-bit, but the source code clearly shows 64-bit support for > measuring rate, so I assume the man page is out of date. So if possible, might also be > worth trying the setup on a newer kernel and version of iproute2 than what's used by > default in the version of Ubuntu mentioned. > > HFSC is also an option as already suggested. I've not used it myself so not familiar > with its configuration or how it compares to HTB+SFQ. But if it doesn't have what > appears to be a 32-bit calculation bug, then that's good. > > Regards, > > Ali > > > On 14/06/2021 17:13, Ethy H. Brito wrote: > > >On Sat, 12 Jun 2021 14:24:21 +0300 > >Anatoly Muliarski <x86ever@xxxxxxxxx> wrote: > > > >>HTB does not work correctly on high speed links. > >>Try to use HFSC. > >I've read some reports about problem of CPU consumption using HFSC+SFQ. > > > >What qdisc do you use associated with each leaf (users) HFSC classes? > > > >Does HFSC inner classes need associated queue disciplines? > > > >Cheers > > > >Ethy > > > >>2021-06-10 1:13 GMT+03:00, Ethy H. Brito <ethy.brito@xxxxxxxxxxxx>: > >>>On Wed, 9 Jun 2021 21:57:30 +0200 > >>>Adam Niescierowicz <adam.niescierowicz@xxxxxxxxxx> wrote: > >>>>W dniu 09.06.2021 o 21:30, Ethy H. Brito pisze: > >>>>>On Wed, 9 Jun 2021 21:11:03 +0200 > >>>>>Adam Niescierowicz <adam.niescierowicz@xxxxxxxxxx> wrote: > >>>>>>How you make filters, it's one level or multilevel based? > >>>>>Multi level. > >>>>> > >>>>>First test IPv4, IPv6 or Other Protocols. > >>>>>Then test for two cidr "/19" and one /20 blocks we use. > >>>>>Then for every /24 block inside the selected cidr block. > >>>>> > >>>>>I can send some fragments of each "test" block, if you want. > >>>>Please send it will be easier. > >>>> > >>>>We have 5+ Gbps at peak, 12core cpu and it's working. We don't use ipv6 > >>>>on this router, but we use PPPoE. > >>>> > >>>> > >>>>What's kernel version?, our is old 4.17. > >>># lsb_release -a > >>>No LSB modules are available. > >>>Distributor ID: Ubuntu > >>>Description: Ubuntu 20.04.2 LTS > >>>Release: 20.04 > >>>Codename: focal > >>> > >>># tc -V > >>>tc utility, iproute2-ss200127 > >>> > >>># uname -a > >>>Linux quiron 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 > >>>x86_64 x86_64 x86_64 GNU/Linux > >>> > >>> > >>>Here we go. > >>> > >>>eth0 is the internal interface (send packets to users - download) > >>>The rules are inserted in the order bellow. There are no inversions. > >>> > >>>Just a very few lines were copied here. > >>>I can send a WeTransfer link for those who may need the full script file. > >>> > >>> ---------------------------8<------------------------------------ > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 1 protocol ipv6 u32 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 2 protocol ip u32 > >>> > >>># Some service networks > >>>/sbin/tc filter add dev eth0 protocol ipv4 parent 1: pref 2 \ > >>> u32 match ip dst 100.64.254.0/24 \ > >>> flowid 1:fffe > >>>/sbin/tc filter add dev eth0 protocol ipv6 parent 1: pref 1 \ > >>> u32 match ip6 dst fe80::/10 \ > >>> flowid 1:fffe > >>> > >>># > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 1: u32 divisor 4 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 10: u32 divisor 256 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 1: match > >>>ip6 dst 2001:12c4:a10:e000::/51 hashkey mask 0x0000ff00 at 28 link 10: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 100: u32 divisor 256 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 10:e0: > >>>match ip6 dst 2001:12c4:a10:e000::/56 hashkey mask 0x000000ff at 28 link > >>>100: > >>> > >>>... links 101 (2001:12c4:a10:e100::/56), 102 (2001:12c4:a10:e200::/56) ... > >>>up to > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 10:ff: > >>>match ip6 dst 2001:12c4:a10:ff00::/56 hashkey mask 0x000000ff at 28 link > >>>11f: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 11: u32 divisor 256 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 1: match > >>>ip6 dst 2001:12c4:6440::/51 hashkey mask 0x0000ff00 at 28 link 11: > >>> > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 120: u32 divisor 256 > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 11:00: > >>>match ip6 dst 2001:12c4:6440:000::/56 hashkey mask 0x000000ff at 28 link > >>>120: > >>> > >>>... same here up to link 13f (2001:12c4:6440:1f00::/56): > >>>More IPv6 filters like these follows to others CIDRs. > >>> > >>>Now for IPv4 /24 blocks. > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 2: u32 divisor 4 > >>> > >>># Hash Table para 256 Blocos /24 de endereços IP para o macro bloco > >>>10.16.224.0/19 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 13: u32 divisor 256 > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 2: match > >>>ip dst 10.16.224.0/19 hashkey mask 0x0000ff00 at 16 link 13: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 150: u32 divisor 256 > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 13:e0: > >>>match ip dst 10.16.224.0/24 hashkey mask 0x000000ff at 16 link 150: > >>> > >>>... link 151 (10.16.225.0/24), 153 (10.16.226.0/24) ... up to > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 16f: u32 divisor 256 > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 13:ff: > >>>match ip dst 10.16.255.0/24 hashkey mask 0x000000ff at 16 link 16f: > >>> > >>>Other CIDRs blocks for 100.64.0/19 follows like... > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 14: u32 divisor 256 > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 2: match > >>>ip dst 100.64.0.0/19 hashkey mask 0x0000ff00 at 16 link 14: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 170: u32 divisor 256 > >>>/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 14:00: > >>>match ip dst 100.64.0.0/24 hashkey mask 0x000000ff at 16 link 170: > >>> > >>>up to link 19f: > >>> > >>>And finally: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 handle ::2 prio 1 protocol ipv6 u32 > >>>match u32 0 0 link 1: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 handle ::2 prio 2 protocol ip u32 > >>>match u32 0 0 link 2: > >>> > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 3 protocol arp u32 match u32 0 > >>>0 classid 1:fffe > >>> > >>># Rede "default" para endereços IPV6 não Classificados anteriormente > >>>/sbin/tc filter add dev eth0 parent 1:0 prio 1 handle ::ffe protocol ipv6 > >>>u32 match u32 0 0 classid 1:fffc > >>> > >>>Thanks for your time. > >>> > >>>Cheers > >>> > >>>Ethy > >> > >>-- > >>Best regards > >>Anatoly Muliarski > > > >
