On 06/12/14 19:32, OmegaPhil wrote: > Disclaimer: I don't do this very often so there is probably a retard > error in here somewhere. I'm not expecting people to do my work for me, > I'm just after a better understanding of the problem so I can get more > control of the situation. > > tldr: Custom priomap + iptables TOS isn't sorting packets correctly, > Wireshark won't even filter on TOS... > > ---- > > I'm currently attempting to implement a 4 band prio shaper with fq_codel > queues on a 100Mbit connection (Debian Testing server): > > ====================================================================== > > tc qdisc add dev eth0 root handle 1: htb default 1 > tc class add dev eth0 parent 1:0 classid 1:1 htb rate 12800kibps ceil > 12800kibps > tc qdisc add dev eth0 parent 1:1 handle 100: prio bands 4 priomap 1 3 1 > 3 2 3 2 3 0 3 0 3 1 3 1 3 > tc qdisc add dev eth0 parent 100:1 handle 1001: fq_codel > tc qdisc add dev eth0 parent 100:2 handle 1002: fq_codel > tc qdisc add dev eth0 parent 100:3 handle 1003: fq_codel > tc qdisc add dev eth0 parent 100:4 handle 1004: fq_codel > > ====================================================================== > > Packets are tagged for the various prio queues via iptables: > > ====================================================================== > > # ICMP > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p icmp -j TOS --set-tos > Minimize-Delay > > # TCP control packets > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK FIN,ACK -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK SYN,ACK -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK RST,ACK -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK RST -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --syn -j TOS --set-tos > Minimize-Delay > > # TCP ACK packets with no or very little data payload (p2p traffic sets > all packets to ACK packets otherwise, source of size: http://phix.me/dm/) > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK ACK -m length --length 40:89 -j TOS --set-tos Minimize-Delay > > # Band 2 prioritisation > # Torrenting > $IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge > -j TOS --set-tos Maximize-Throughput > > # Band 3 prioritisation > #$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user1 > -j TOS --set-tos Minimize-Cost > #$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user2 > -j TOS --set-tos Minimize-Cost > > ====================================================================== > > This is based on an otherwise-successful configuration on a local Ubuntu > server that admittedly doesn't originate traffic itself, without a > custom priomap. > > The general idea is: > > Band 0: High priority TCP packets, Minimize Delay, > Band 1: Normal (nothing targetted here) > Band 2: Torrenting, Maximize Throughput > Band 3: Special programs, Minimize Monetary Cost > > When I let the above run, virtually all packets get dumped into band 1, > whereas by far the bulk of the traffic is torrenting - the shaping code > is behaving like iptables isn't tagging the packets properly, however > 'iptables -v -L -t mangle' is showing a lot of packets going through the > TOS rules. > > I next captured packets and opened up with Wireshark to see what was > going on (it would be nice if I could just capture from the queues > directly but I've found no evidence this is possible), however the > following expressions fail to return anything: > > ip.tos > ip.tos==8 > ip.tos==0x8 > > etc with other values - I then moved to ip.dsfield.dscp, this failed in > a different way - ip.dsfield.dscp==2 returned packets with > 'Differentiated Services Field: 0x08', ip.dsfield.dscp==2 returned 0x10 > - why? > > At this point I stopped as I clearly didn't know what I was doing. Any > pointers? > > Thanks for any help. This answering my own question for others that want a simple strict priority hierarchy with a customisable band count: I've finally managed to get a custom number of bands PRIO queue on my server working now (no need to maintain a custom kernel, tc etc) - the key was to drop the broken TOS classification and just the iptables CLASSIFY target directly (no need to get involved in complicated tc filter stuff either): Aim: Band 0: SSH traffic Band 1: 'Normal' traffic, anything unclassified including iroffer Band 2: Torrent traffic Band 3: Darknet traffic Setup 4 band PRIO qdisc: ======================================================================= tc qdisc add dev eth0 parent root handle 1: prio bands 4 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ======================================================================= Handle must be 1+, it doesn't like 0, you end up with a 8000+ number that naturally breaks any later references in iptables. Note that the band number in priomap counts from 0, so the bands are 0, 1, 2 and 3 - the actual qdisc IDs start from 1 (...). Dumping in band 1 (band 2 qdisc ID) across the board acts as the default classification. Setup usual fq_codel qdiscs: ======================================================================= tc qdisc add dev eth0 parent 1:1 handle 101: fq_codel tc qdisc add dev eth0 parent 1:2 handle 102: fq_codel tc qdisc add dev eth0 parent 1:3 handle 103: fq_codel tc qdisc add dev eth0 parent 1:4 handle 104: fq_codel ======================================================================= The child PRIO qdiscs associated with your bands have been created for you already and their ID starts from 1. Now get iptables to do the classification: SSH (port 22222 here): ======================================================================= iptables -t mangle -A POSTROUTING -o eth0 -p tcp -s "$PUBLIC_IP" --sport 22222 -j CLASSIFY --set-class 1:1 ======================================================================= Torrenting: ======================================================================= iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j CLASSIFY --set-class 1:3 ======================================================================= Darknets: ======================================================================= iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner debian-tor -j CLASSIFY --set-class 1:4 iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner i2p -j CLASSIFY --set-class 1:4 ======================================================================= Everything else ends up in 1:2 as mentioned previously due to the initial priomap. For a nice realtime view of how packets are flowing through the qdiscs to prove things are actually doing what you told them to do, use bmon (https://github.com/tgraf/bmon) - literally the 'bmon' command, then move the left white cursor thing up and down to select the interface or qdisc/class you are interested in.
Attachment:
signature.asc
Description: OpenPGP digital signature
