Hi,
The initial query mentioned the issue occurs at around 4.5Gbps throughput or higher.
I wonder if a 32-bit value is overflowing somewhere. If the traffic rate or some other
value used for internal calculations of the qdisc uses an unsigned 32-bit value, then
this can only go up to 4294967295 which is just short of 4.3Gbps. So would make sense
problems start to occur when you go beyond this throughput rate. I don't think the
filters or quantity of filters would be the problem, they just classify the traffic.
The problem would likely be within how a qdisc is implemented and how and when it
chooses to dequeue packets for transmission. More likely the bug is within HTB than
SFQ, as I don't think SFQ cares about throughput rate, but that's just my guess.
How are the qdiscs/classes created? It would be interesting to compare with the
configuration from the person who has it working at 5Gbps.
I also thought tc/iproute2 supported 64-bit values for traffic rate. The tc man page
says otherwise i.e. only 32-bit, but the source code clearly shows 64-bit support for
measuring rate, so I assume the man page is out of date. So if possible, might also be
worth trying the setup on a newer kernel and version of iproute2 than what's used by
default in the version of Ubuntu mentioned.
HFSC is also an option as already suggested. I've not used it myself so not familiar
with its configuration or how it compares to HTB+SFQ. But if it doesn't have what
appears to be a 32-bit calculation bug, then that's good.
Regards,
Ali
On 14/06/2021 17:13, Ethy H. Brito wrote:
On Sat, 12 Jun 2021 14:24:21 +0300
Anatoly Muliarski <x86ever@xxxxxxxxx> wrote:
HTB does not work correctly on high speed links.
Try to use HFSC.
I've read some reports about problem of CPU consumption using HFSC+SFQ.
What qdisc do you use associated with each leaf (users) HFSC classes?
Does HFSC inner classes need associated queue disciplines?
Cheers
Ethy
2021-06-10 1:13 GMT+03:00, Ethy H. Brito <ethy.brito@xxxxxxxxxxxx>:
On Wed, 9 Jun 2021 21:57:30 +0200
Adam Niescierowicz <adam.niescierowicz@xxxxxxxxxx> wrote:
W dniu 09.06.2021 o 21:30, Ethy H. Brito pisze:
On Wed, 9 Jun 2021 21:11:03 +0200
Adam Niescierowicz <adam.niescierowicz@xxxxxxxxxx> wrote:
How you make filters, it's one level or multilevel based?
Multi level.
First test IPv4, IPv6 or Other Protocols.
Then test for two cidr "/19" and one /20 blocks we use.
Then for every /24 block inside the selected cidr block.
I can send some fragments of each "test" block, if you want.
Please send it will be easier.
We have 5+ Gbps at peak, 12core cpu and it's working. We don't use ipv6
on this router, but we use PPPoE.
What's kernel version?, our is old 4.17.
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
# tc -V
tc utility, iproute2-ss200127
# uname -a
Linux quiron 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021
x86_64 x86_64 x86_64 GNU/Linux
Here we go.
eth0 is the internal interface (send packets to users - download)
The rules are inserted in the order bellow. There are no inversions.
Just a very few lines were copied here.
I can send a WeTransfer link for those who may need the full script file.
---------------------------8<------------------------------------
/sbin/tc filter add dev eth0 parent 1:0 prio 1 protocol ipv6 u32
/sbin/tc filter add dev eth0 parent 1:0 prio 2 protocol ip u32
# Some service networks
/sbin/tc filter add dev eth0 protocol ipv4 parent 1: pref 2 \
u32 match ip dst 100.64.254.0/24 \
flowid 1:fffe
/sbin/tc filter add dev eth0 protocol ipv6 parent 1: pref 1 \
u32 match ip6 dst fe80::/10 \
flowid 1:fffe
#
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 1: u32 divisor 4
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 10: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 1: match
ip6 dst 2001:12c4:a10:e000::/51 hashkey mask 0x0000ff00 at 28 link 10:
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 100: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 10:e0:
match ip6 dst 2001:12c4:a10:e000::/56 hashkey mask 0x000000ff at 28 link
100:
... links 101 (2001:12c4:a10:e100::/56), 102 (2001:12c4:a10:e200::/56) ...
up to
/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 10:ff:
match ip6 dst 2001:12c4:a10:ff00::/56 hashkey mask 0x000000ff at 28 link
11f:
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 11: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 1: match
ip6 dst 2001:12c4:6440::/51 hashkey mask 0x0000ff00 at 28 link 11:
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 120: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 ht 11:00:
match ip6 dst 2001:12c4:6440:000::/56 hashkey mask 0x000000ff at 28 link
120:
... same here up to link 13f (2001:12c4:6440:1f00::/56):
More IPv6 filters like these follows to others CIDRs.
Now for IPv4 /24 blocks.
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 2: u32 divisor 4
# Hash Table para 256 Blocos /24 de endereços IP para o macro bloco
10.16.224.0/19
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 13: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 2: match
ip dst 10.16.224.0/19 hashkey mask 0x0000ff00 at 16 link 13:
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 150: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 13:e0:
match ip dst 10.16.224.0/24 hashkey mask 0x000000ff at 16 link 150:
... link 151 (10.16.225.0/24), 153 (10.16.226.0/24) ... up to
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 16f: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 13:ff:
match ip dst 10.16.255.0/24 hashkey mask 0x000000ff at 16 link 16f:
Other CIDRs blocks for 100.64.0/19 follows like...
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 14: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 2: match
ip dst 100.64.0.0/19 hashkey mask 0x0000ff00 at 16 link 14:
/sbin/tc filter add dev eth0 parent 1:0 prio 99 handle 170: u32 divisor 256
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 ht 14:00:
match ip dst 100.64.0.0/24 hashkey mask 0x000000ff at 16 link 170:
up to link 19f:
And finally:
/sbin/tc filter add dev eth0 parent 1:0 handle ::2 prio 1 protocol ipv6 u32
match u32 0 0 link 1:
/sbin/tc filter add dev eth0 parent 1:0 handle ::2 prio 2 protocol ip u32
match u32 0 0 link 2:
/sbin/tc filter add dev eth0 parent 1:0 prio 3 protocol arp u32 match u32 0
0 classid 1:fffe
# Rede "default" para endereços IPV6 não Classificados anteriormente
/sbin/tc filter add dev eth0 parent 1:0 prio 1 handle ::ffe protocol ipv6
u32 match u32 0 0 classid 1:fffc
Thanks for your time.
Cheers
Ethy
--
Best regards
Anatoly Muliarski
