Hi Andy.
Thanks !! Thanks a lot for your help !!!!
Finally I'm seeing something working in the right way... ;)
I?ve tried another sender/receiver program (I find the ?netperf? a little difficult to operate), so I?ve tried with
?iperf?.
Now I?m getting all the flows with the correct limitation for its corresponding class.
But also now, I?m getting ONLY the RATE limitation, not the CEIL one !!! :(
I?ve done a smaller test configuration, with the same 2 PCs, connected the same way and added the following rules:
======================================================
#interface="eth0"
#interface="lo"
interface="ppp0"
======================================================
#interface="eth0"
#interface="lo"
interface="ppp0"
#Delete any previous stored configuration
tc qdisc del dev $interface root
tc qdisc del dev $interface root
#Creating the root Qdisc (Queueing Disk)
tc qdisc add dev $interface root handle 1: htb default 20
tc qdisc add dev $interface root handle 1: htb default 20
#Definition of classes
tc class add dev $interface parent 1: classid 1:1 htb rate 28kbit ceil 28kbit
tc class add dev $interface parent 1:2 classid 1:20 htb rate 10kbit ceil 28kbit prio 0
tc class add dev $interface parent 1:3 classid 1:30 htb rate 15kbit ceil 28kbit prio 1
tc class add dev $interface parent 1: classid 1:1 htb rate 28kbit ceil 28kbit
tc class add dev $interface parent 1:2 classid 1:20 htb rate 10kbit ceil 28kbit prio 0
tc class add dev $interface parent 1:3 classid 1:30 htb rate 15kbit ceil 28kbit prio 1
tc qdisc add dev $interface parent 1:20 handle 30: sfq
tc qdisc add dev $interface parent 1:30 handle 40: sfq
tc qdisc add dev $interface parent 1:30 handle 40: sfq
#Definition of the filters
tc filter add dev $interface protocol ip parent 1:0 u32 match ip dport 20001 0xffff flowid 1:20
tc filter add dev $interface protocol ip parent 1:0 u32 match ip dport 20002 0xffff flowid 1:30
==========================================================
tc filter add dev $interface protocol ip parent 1:0 u32 match ip dport 20001 0xffff flowid 1:20
tc filter add dev $interface protocol ip parent 1:0 u32 match ip dport 20002 0xffff flowid 1:30
==========================================================
This is my actual version of HTB:
==========================================================
[root@EdenRH9 htb]# tc qdisc add htb help
What is "help"?
Usage: ... qdisc add ... htb [default N] [r2q N]
default minor id of class to which unclassified packets are sent {0}
r2q DRR quantums are computed as rate in Bps/r2q {10}
debug string of 16 numbers each 0-3 {0}
[root@EdenRH9 htb]# tc qdisc add htb help
What is "help"?
Usage: ... qdisc add ... htb [default N] [r2q N]
default minor id of class to which unclassified packets are sent {0}
r2q DRR quantums are computed as rate in Bps/r2q {10}
debug string of 16 numbers each 0-3 {0}
... class add ... htb rate R1 burst B1 [prio P] [slot S] [pslot PS]
[ceil R2] [cburst B2] [mtu MTU] [quantum Q]
rate rate allocated to this class (class can still borrow)
burst max bytes burst which can be accumulated during idle period {computed}
ceil definite upper class rate (no borrows) {rate}
cburst burst but for ceil {computed}
mtu max packet size we create rate map for {1600}
prio priority of leaf; lower are served first {0}
quantum how much bytes to serve from leaf at once {use r2q}
[ceil R2] [cburst B2] [mtu MTU] [quantum Q]
rate rate allocated to this class (class can still borrow)
burst max bytes burst which can be accumulated during idle period {computed}
ceil definite upper class rate (no borrows) {rate}
cburst burst but for ceil {computed}
mtu max packet size we create rate map for {1600}
prio priority of leaf; lower are served first {0}
quantum how much bytes to serve from leaf at once {use r2q}
TC HTB version 3.3
[root@EdenRH9 htb]#
==========================================================
[root@EdenRH9 htb]#
==========================================================
When I did a test run with the traffic generator Iperf, utilizing the following command I got the following results:
==========================================================
[root@EdenRH9 htb]# iperf -c 192.168.7.100 -p 20001 -n 1 -l 50000
------------------------------------------------------------
Client connecting to 192.168.7.100, TCP port 20001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.7.200 port 35550 connected with 192.168.7.100 port 20001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-37.4 sec 48.8 KBytes 10.7 Kbits/sec
[root@EdenRH9 htb]#
==========================================================
[root@EdenRH9 htb]# iperf -c 192.168.7.100 -p 20001 -n 1 -l 50000
------------------------------------------------------------
Client connecting to 192.168.7.100, TCP port 20001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.7.200 port 35550 connected with 192.168.7.100 port 20001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-37.4 sec 48.8 KBytes 10.7 Kbits/sec
[root@EdenRH9 htb]#
==========================================================
You can already see that the rate didn?t went up to the 28kbit, and there was no other traffic on that interface or other class.
So I?ve printed the output of the TC command regarding its QDISC, CLASS and FILTER, before the test run and after the test run and got the following answers:
So I?ve printed the output of the TC command regarding its QDISC, CLASS and FILTER, before the test run and after the test run and got the following answers:
==========================================================
[root@EdenRH9 htb]# tc -s -d class show dev ppp0
class htb 1:1 root prio 0 quantum 1000 rate 28Kbit ceil 28Kbit burst 1634b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 373714 ctokens: 373714
[root@EdenRH9 htb]# tc -s -d class show dev ppp0
class htb 1:1 root prio 0 quantum 1000 rate 28Kbit ceil 28Kbit burst 1634b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 373714 ctokens: 373714
class htb 1:20 root leaf 30: prio 0 quantum 1000 rate 10Kbit ceil 28Kbit burst 1611b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 1031680 ctokens: 373714
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 1031680 ctokens: 373714
class htb 1:30 root leaf 40: prio 1 quantum 1000 rate 15Kbit ceil 28Kbit burst 1618b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 690773 ctokens: 373714
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 690773 ctokens: 373714
[root@EdenRH9 htb]# tc -s -d filter show dev ppp0
filter parent 1: protocol ip pref 49151 u32
filter parent 1: protocol ip pref 49151 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49151 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32
filter parent 1: protocol ip pref 49152 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
[root@EdenRH9 htb]# tc -s -d qdisc show dev ppp0
qdisc sfq 40: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter parent 1: protocol ip pref 49151 u32
filter parent 1: protocol ip pref 49151 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49151 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32
filter parent 1: protocol ip pref 49152 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
[root@EdenRH9 htb]# tc -s -d qdisc show dev ppp0
qdisc sfq 40: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc sfq 30: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc htb 1: r2q 10 default 20 direct_packets_stat 0 ver 3.7
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]# tc -s -d qdisc show dev ppp0
qdisc sfq 40: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]#
[root@EdenRH9 htb]# tc -s -d qdisc show dev ppp0
qdisc sfq 40: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc sfq 30: quantum 1500b limit 128p flows 128/1024
Sent 52060 bytes 39 pkts (dropped 0, overlimits 0)
Sent 52060 bytes 39 pkts (dropped 0, overlimits 0)
qdisc htb 1: r2q 10 default 20 direct_packets_stat 0 ver 3.7
Sent 52060 bytes 39 pkts (dropped 0, overlimits 65)
Sent 52060 bytes 39 pkts (dropped 0, overlimits 65)
[root@EdenRH9 htb]# tc -s -d filter show dev ppp0
filter parent 1: protocol ip pref 49151 u32
filter parent 1: protocol ip pref 49151 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49151 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32
filter parent 1: protocol ip pref 49152 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
[root@EdenRH9 htb]# tc -s -d class show dev ppp0
class htb 1:1 root prio 0 quantum 1000 rate 28Kbit ceil 28Kbit burst 1634b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 373714 ctokens: 373714
filter parent 1: protocol ip pref 49151 u32
filter parent 1: protocol ip pref 49151 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49151 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32
filter parent 1: protocol ip pref 49152 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:30
match 00004e22/0000ffff at 20
filter parent 1: protocol ip pref 49152 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:20
match 00004e21/0000ffff at 20
[root@EdenRH9 htb]# tc -s -d class show dev ppp0
class htb 1:1 root prio 0 quantum 1000 rate 28Kbit ceil 28Kbit burst 1634b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 373714 ctokens: 373714
class htb 1:20 root leaf 30: prio 0 quantum 1000 rate 10Kbit ceil 28Kbit burst 1611b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 52060 bytes 39 pkts (dropped 0, overlimits 0)
rate 131bps
lended: 39 borrowed: 0 giants: 0
tokens: -25088 ctokens: 362744
Sent 52060 bytes 39 pkts (dropped 0, overlimits 0)
rate 131bps
lended: 39 borrowed: 0 giants: 0
tokens: -25088 ctokens: 362744
class htb 1:30 root leaf 40: prio 1 quantum 1000 rate 15Kbit ceil 28Kbit burst 1618b/8 mpu 0b cburst 1634b/8 mpu 0b level 0
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 690773 ctokens: 373714
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 690773 ctokens: 373714
[root@EdenRH9 htb]#
==========================================================
==========================================================
Also made a printout from the TCPTRACE output, captured on the other PC with ETHEREAL with the following content:
================================================
1 arg remaining, starting with '/opt/teste_regras20001.ethereal'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
1 arg remaining, starting with '/opt/teste_regras20001.ethereal'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
76 packets seen, 76 TCP packets traced
elapsed wallclock time: 0:00:00.006178, 12301 pkts/sec analyzed
trace file elapsed time: 0:00:39.289997
TCP connection info:
1 TCP connection traced:
TCP connection 1:
host a: 192.168.7.200:35550
host b: 192.168.7.100:20001
complete conn: yes
first packet: Sun Oct 23 20:26:51.304505 2005
last packet: Sun Oct 23 20:27:30.594503 2005
elapsed time: 0:00:39.289997
total packets: 76
filename: /opt/teste_regras20001.ethereal
a->b: b->a:
total packets: 39 total packets: 37
ack pkts sent: 38 ack pkts sent: 37
pure acks sent: 2 pure acks sent: 35
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 50024 unique bytes sent: 0
actual data pkts: 36 actual data pkts: 0
actual data bytes: 50024 actual data bytes: 0
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 10 pushed data pkts: 0
SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/1
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 0 adv wind scale: 0
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 1448 bytes max segm size: 0 bytes
min segm size: 24 bytes min segm size: 0 bytes
avg segm size: 1389 bytes avg segm size: 0 bytes
max win adv: 5840 bytes max win adv: 63712 bytes
min win adv: 5840 bytes min win adv: 5792 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 5840 bytes avg win adv: 46818 bytes
initial window: 24 bytes initial window: 0 bytes
initial window: 1 pkts initial window: 0 pkts
ttl stream length: 50024 bytes ttl stream length: 0 bytes
missed data: 0 bytes missed data: 0 bytes
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 37.430 secs data xmit time: 0.000 secs
idletime max: 2370.0 ms idletime max: 2374.2 ms
throughput: 1273 Bps throughput: 0 Bps
=============================================================
elapsed wallclock time: 0:00:00.006178, 12301 pkts/sec analyzed
trace file elapsed time: 0:00:39.289997
TCP connection info:
1 TCP connection traced:
TCP connection 1:
host a: 192.168.7.200:35550
host b: 192.168.7.100:20001
complete conn: yes
first packet: Sun Oct 23 20:26:51.304505 2005
last packet: Sun Oct 23 20:27:30.594503 2005
elapsed time: 0:00:39.289997
total packets: 76
filename: /opt/teste_regras20001.ethereal
a->b: b->a:
total packets: 39 total packets: 37
ack pkts sent: 38 ack pkts sent: 37
pure acks sent: 2 pure acks sent: 35
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 50024 unique bytes sent: 0
actual data pkts: 36 actual data pkts: 0
actual data bytes: 50024 actual data bytes: 0
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 10 pushed data pkts: 0
SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/1
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 0 adv wind scale: 0
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 1448 bytes max segm size: 0 bytes
min segm size: 24 bytes min segm size: 0 bytes
avg segm size: 1389 bytes avg segm size: 0 bytes
max win adv: 5840 bytes max win adv: 63712 bytes
min win adv: 5840 bytes min win adv: 5792 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 5840 bytes avg win adv: 46818 bytes
initial window: 24 bytes initial window: 0 bytes
initial window: 1 pkts initial window: 0 pkts
ttl stream length: 50024 bytes ttl stream length: 0 bytes
missed data: 0 bytes missed data: 0 bytes
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 37.430 secs data xmit time: 0.000 secs
idletime max: 2370.0 ms idletime max: 2374.2 ms
throughput: 1273 Bps throughput: 0 Bps
=============================================================
Why are the rules for RATE being the only ones that are taken into account for bandwidth limitation and HTB does not care about the CEIL in class (1:20) and borrows tokens from the other class (1:30) that?s not being utilized?
Can I still have a problem with software versions (Linux, HTB or TC) ?
Can there be a problem of correct shaping in HTB for smaller bitrates ?
Thanks in advance.
Best regards.
Paulo AUgusto
From: Andy Furniss <andy.furniss@xxxxxxxxxxxxx>
Reply-To: andy.furniss@xxxxxxxxxxxxx
To: Paulo Augusto <augustopaulo@xxxxxxxxxxx>
CC: lartc@xxxxxxxxxxxxxxx
Subject: Re: Lost packets and strange "behaviour" of my TC rules
Date: Mon, 17 Oct 2005 22:29:12 +0100
>Paulo Augusto wrote:
>
>
>>
>>I'm using the TG (www.postel.org/tg <http://www.postel.org/tg>) as
>>a TCP traffic generator, to establish three 90kbits/s TCP flows
>>from PC1(any port) to PC2(port 20000, 20001 and 20002), with
>>different durations and pause times, which as can be shown in the
>>next files:
>
>Usually tcp won't be like this, netperf may be better to test with,
>as it's more normal for bulk tcp to try to go as fast as it can.
>
>>
>>I've also plotted a graph with gnuplot, showing me that (somehow)
>>my rules were "correct", I've got an average value of 30kbits/s, a
>>10kbits/s and a 60kbits/s.
>>
>>Is it normal that some packets get dropped by the rules or not
>>(taking into account my test configuration) ?
>
>The packets are not dropped in this case as the default queue length
>for sfq is 128, which can hold a rwin worth of data. The missing
>packets just didn't get sent because htb slowed the packets down to
>the rates specified and the sender will only send more once the ones
>already sent are acked.
>
>Andy.
>
_______________________________________________ LARTC mailing list LARTC@xxxxxxxxxxxxxxx http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
